Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.0     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.2     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Introduction

When you have two categorical variables, and you want to test for an association between them, the go-to method is a chi-squared test. For example, you might have a survey of eyewear worn by people who classify themselves as male or female:

my_url <- "http://ritsokiguess.site/datafiles/eyewear.txt"
eyewear <- read_delim(my_url, " ")

Rows: 2 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: " "
chr (1): gender
dbl (3): contacts, glasses, none

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

eyewear

Is there a tendency for one gender to prefer a certain type of eyewear over the other, or, if you know a person’s gender, can you predict what they are likely to wear on their eyes? That is to say, is there an association between eyewear and gender?

eyewear %>% 
  select(-gender) %>% 
  as.matrix() %>% 
  chisq.test() -> eyewear_chisq
eyewear_chisq

    Pearson's Chi-squared test

data:  .
X-squared = 17.718, df = 2, p-value = 0.0001421

With a P-value of 0.0001, there certainly is an association.

The above is the way contingency tables are usually laid out, but is not the most convenient for graphs (and, as we see later, for log-linear modelling):

eyewear %>% 
  pivot_longer(-gender, names_to = "eyewear", values_to = "count") -> eyewear1
eyewear1

Now we can draw a graph to see what kind of association there is:

ggplot(eyewear1, aes(x = gender, y = count, fill = eyewear)) + 
  geom_col(position = "fill")

Females are more likely to wear contacts and less likely to wear glasses than males.

A word about the graph: this is a bar chart, but one with two properties:

• the frequencies are not counted from the data, but given in a variable, so we use geom_col rather than geom_bar. In geom_col, the y aesthetic contains the frequencies.
• the bars of the bar chart are stacked, but also scaled to have unit height (by dividing by the totals for that bar). The -axis label count is thus a misnomer; they are proportions rather than counts.

In a contingency table, one of the variables is often logically an outcome, eyewear here. Putting the logical explanatory variable as x and the logical outcome as fill enables us to compare the size of the coloured rectangles for each value of the variable on the -axis (which is the appropriate way to do it).¹

Log-linear models

For two categorical variables, a chi-squared test followed by a graph (if the result is significant, to understand the association) is a straightforward way to go.

But what if we have more than two categorical variables? The picture can be more nuanced, in that there can be associations between only some of the categorical variables, and we now want to do two things: first, find out which categorical variables are associated, and second, for those that are, understand those associations.

The second thing can be done by drawing a graph like the one above, but the first needs a new modelling procedure: a log-linear model. This models the frequencies as having a Poisson distribution, with the log of the frequency having a linear relationship with effects for each categorical variable and possibly their interactions. A log-linear model is a generalized linear model with Poisson family (using the default log link).

One point that can be tricky to understand is that one or more of the categorical variables can be logically a response (like eyewear above), but as far as the modelling is concerned, all categorical variables are explanatory, with the frequency variable being the response.

The starting point is to fit a model including all interactions between (model) explanatory variables, with the frequency variable as (model) response. Let’s illustrate with the eyewear data, using the “long” layout (which is needed for log-linear modelling):

eyewear1.1 <- glm(count ~ gender * eyewear,
                  family = "poisson",
                  data = eyewear1)

The next step is to see what can be removed from this model, which is most easily done with drop1:²

drop1(eyewear1.1, test = "Chisq")

The interaction between gender and eyewear is significant, and the P-value is close to the one from the chi-squared test. In this model, this indicates a significant association between the categorical variables whose interaction is significant (in other words, the same conclusion as for the chi-squared test, with almost the same P-value).

Technical aside: the P-values from the log-linear model and from the chi-squared test are very close, but not exactly equal. Letting denote an observed frequency and the corresponding expected frequency, the chi-squared test statistic is the sum over all cells of , the so-called Pearson chi-squared, while the log-linear test statistic in this case is the sum of , the likelihood ratio test statistic. These are not the same but are often very similar. End of technical aside.

Which airline to fly?

Let’s do a more interesting example. Two airlines, Alaska Airlines and America West, fly into various airports in the west of the US. Each flight is either on time or delayed at its destination. Is one airline more punctual than the other, and does this depend on which airport you are looking at? This time we have three categorical variables: airline, airport, and on time/delayed, which we will call status. I found these data in a textbook, laid out like this:

               Alaska Airlines       America West
Airport       On time    Delayed   On time    Delayed
Los Angeles      497        62        694       117
Phoenix          221        12       4840       415
San Diego        212        20        383        65
San Francisco    503       102        320       129
Seattle         1841       305        201        61

Total           3274       501       6438       787

This is not modelling-friendly, and not even tidy (in the tidyverse sense of the term). There is a further problem in that we have two rows of headers (often the way a three way table is displayed). I condensed the two rows of headers into one:

airport    aa_ontime aa_delayed aw_ontime aw_delayed
LosAngeles   497          62       694        117
Phoenix      221          12      4840        415
SanDiego     212          20       383         65
SanFrancisco 503         102       320        129
Seattle     1841         305       201         61

which poses an additional problem in a moment, but at least we can now read in the data:

my_url <- "http://ritsokiguess.site/datafiles/airlines.txt"
airlines <- read_table(my_url)

── Column specification ────────────────────────────────────────────────────────
cols(
  airport = col_character(),
  aa_ontime = col_double(),
  aa_delayed = col_double(),
  aw_ontime = col_double(),
  aw_delayed = col_double()
)

airlines

We need to get each of the three categorical variables in its own column, with one column of frequencies. This uses the variant of pivot_longer with two variable names to go into names_to, separated by an underscore:

airlines %>%
   pivot_longer(-airport, 
                names_to = c("airline", "status"), 
                names_sep = "_", 
                values_to = "freq" ) -> punctual
punctual

and now we are ready to go.

The first stage is a log-linear model with the three-way interaction between airport, airline, and status, using freq as the (model) response:

punctual.1 <- glm(freq ~ airport * airline * status,
                  family = "poisson",
                  data = punctual)

and then we see what if anything we can get rid of. drop1 is nicer than summary for two reasons: (i) it only displays what actually can be dropped (the highest order interactions in this kind of model), (ii) when a categorical variable has more than two levels (such as airport here), summary only shows how each level compares to the baseline, not whether there is an effect of that categorical variable as a whole.

So:

drop1(punctual.1, test = "Chisq")

The three-way interaction is not significant, so that term can be removed from the model. I like update for this task, because the model is complicated enough that copying, pasting, and editing the previous glm code risks fitting the wrong model next:³

punctual.2 <- update(punctual.1, . ~ . - airport:airline:status)
drop1(punctual.2, test = "Chisq")

There are three two-way interactions up for grabs, but they are all significant, so this is where we stop. The modelling indicates there are significant associations between airport and airline, airport and status, airline and status.

As far as the modelling is concerned, all three categorical variables are “explanatory”, but from a logical point of view, status seems like an outcome: particular airlines or airports cause a flight to be on time or delayed. Our main interest in this kind of modelling is “what is associated with the logical outcome?”. The log-linear modelling says that there is an effect of airline on status, and also a separate effect of airport on status (but not an effect of the combination of airline and airport on status, because the airport:airline:status interaction was not significant).

The airline - status association was what interested us most, so let’s go after that, remembering that in our graphs, the logical outcome should be fill:

ggplot(punctual, aes(x = airline, y = freq, fill = status)) +
  geom_col(position = "fill")

This is a bit hard to see, so let’s zoom in on the -axis:

ggplot(punctual, aes(x = airline, y = freq, fill = status)) + 
  geom_col(position = "fill") +
  coord_cartesian(ylim = c(0.75, 1))

Figure 1 says that Alaska Airlines is a bit less punctual than America West overall, about 87% on time vs. about 89% on time. So, we seem to have an answer to our question. But we have two other significant associations to explore. Bear in mind that the graph above is “collapsed” over airports.

The next most interesting association is the other one with status, namely airport by status:

ggplot(punctual, aes(x = airport, y = freq, fill = status)) +
  geom_col(position = "fill")

Figure 2 says that the airports differ a fair bit in terms of punctuality, from over 95% on time at Phoenix down to only just over 75% at San Francisco.⁴

So, the airports are different in terms of punctuality, but the status-airline graph above aggregates over the different airports. Some of you may be beginning to feel uneasy at this point.

It is usually only associations with the logical outcome variable that concern us, but we have one more association that we can investigate, airport by airline. Neither of these are logical outcomes, so it doesn’t matter which is x and which is fill, but there are more airports than airlines, so I’ll put airports on the -axis:

ggplot(punctual, aes(x = airport, y = freq, fill = airline)) +
  geom_col(position = "fill")

The airports are actually very different in terms of which of the airlines fly into them: almost all of the flights into Phoenix are America West, and most of the flights into San Francisco and Seattle are Alaska Airlines.

Figure 2 and Figure 3 tell us that there is a potential bias here: America West fly mostly into Phoenix, where it is easier to be on time, and Alaska Airlines fly mostly into Seattle and San Francisco, where it is more difficult to be on time. Maybe the punctuality picture for Alaska Airlines is better than Figure 1 suggests?

Let’s compare apples with apples: make a graph of the punctuality of each airline at each airport, since the airports do differ one from another:

ggplot(punctual, aes(x = airline, y = freq, fill = status)) +
  geom_col(position = "fill") + facet_wrap(~ airport)

Once again, zooming in on the -axis seems to be the thing:

ggplot(punctual, aes(x = airline, y = freq, fill = status)) +
  geom_col(position = "fill") + facet_wrap(~ airport) +
  coord_cartesian(ylim = c(0.6, 1))

and now we see that Alaska Airlines is more punctual at every single airport than America West, despite being less punctual overall! In other words, we have unearthed a Simpson’s Paradox.

Figure 4 is the kind of thing we would draw to investigate a three-way association. In this case, airport:airline:status was not significant so we didn’t look at this graph before. The kind of thing that would have made the three-way association significant is if one airline was more punctual than the other at some airports and less punctual at others. Here, Alaska Airlines is more punctual at every airport, and, in Figure 4, by about the same amount everywhere, so that the three-way interaction was not significant. What prompted us to look at this graph was the potential for bias by aggregating over airports in Figure 1 that seemed to be very different from each other in Figure 2 and Figure 3. That small difference in punctuality between airlines in Figure 1 was significant, but not in the way we thought!

Ovarian cancer, a four-way table

In a 1973 retrospective study, 299 women who had previously been operated on for ovarian cancer were assessed, and four variables measured:

• stage of cancer at time of operation (early or advanced)
• type of operation (radical or limited)
• X-ray treatment received (xray) (yes or no)
• 10-year survival (yes or no)

Logically, survival is the outcome, so we will be mainly interested in associations with that.

The data, no tidying needed this time:

my_url <- "http://ritsokiguess.site/datafiles/cancer.txt"
cancer <- read_delim(my_url, " ")

Rows: 16 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: " "
chr (4): stage, operation, xray, survival
dbl (1): freq

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

cancer 

Strap yourself in. Step 1:

cancer.1 <- glm(freq ~ stage * operation * xray * survival,
  data = cancer, family = "poisson")
drop1(cancer.1, test = "Chisq")

The four-way interaction (thankfully) comes out:

cancer.2 <- update(cancer.1, 
                   . ~ . - stage:operation:xray:survival)
drop1(cancer.2, test = "Chisq")

None of the three-way interactions are significant (at least in this model), so we remove the least significant of them:

cancer.3 <- update(cancer.2, . ~ . - stage:xray:survival)
drop1(cancer.3, test = "Chisq")

and none of these are significant, so remove the least significant of them:

cancer.4 <- update(cancer.3, . ~ . - operation:xray:survival)
drop1(cancer.4, test = "Chisq")

Now there is a two-way interaction up for grabs, and finally something is significant. The two-way interaction has the highest P-value, so out it comes:

cancer.5 <- update(cancer.4, . ~ . - xray:survival)
drop1(cancer.5, test = "Chisq")

and on we go:

cancer.6 <- update(cancer.5, . ~ . - stage:operation:survival)
drop1(cancer.6, test = "Chisq")

It looks as if we are beginning to get somewhere. One non-significant effect here:

cancer.7 <- update(cancer.6, . ~ . - operation:survival)
drop1(cancer.7, test = "Chisq")

and now everything is significant, so we breathe a sigh of relief and see what’s left. Out of the two remaining significant associations, only one involves our logical outcome survival, and that is stage:survival:

ggplot(cancer, aes(x = stage, y = freq, fill = survival)) + 
  geom_col(position = "fill")

that is to say, most of the patients whose cancer was in the early stage survived for 10 years, and most of those whose cancer was in the advanced stage did not.⁵

The people who collected these data were probably looking for an effect of the actual treatments, but both xray:survival and operation:survival disappeared from our model by virtue of not being significant, so there is no evidence of either the X-ray treatment or the type of operation having any effect on survival.

That other association is of less interest, but we can still look at it:

ggplot(cancer, aes(x = stage, y = freq, fill = xray)) + 
  geom_col(position = "fill") + facet_wrap(~ operation)

If a patient had the limited operation, whether or not they also had the X-ray treatment depended on the stage of cancer (more likely if it was advanced). On the other hand, if the patient had the radical operation, whether or not they had the X-ray treatment had nothing to do with the stage of cancer.

This last, of course, has nothing to do with survival.

Introduction

In Part 1, I talked about sharing your Quarto website with the world using Codeberg Pages. But, what if you have your own domain, and you want your website to appear there? My recommendation is to use the service grebedoc.dev. There is some setup involved, but I will take you through that.

One big website or several small ones?

Before we get to details, it is worth making a design decision: do you want one website that grows lots of pieces as your site expands, or do you want several small websites that you can link together?

I went the first way, but eventually came to regret my decision, because the big website had to be re-rendered as a whole every time I made a change anywhere, and that began to take a while. I teach at a university, so my website naturally has one part for each course I teach, as well as one for the program I coordinate. This was originally one Quarto website, with each part in a subfolder. This works perfectly well, but, as I say, was starting to get unwieldy, especially given that I was typically making changes to only one part of it at a time.

Andrew Heiss teaches at Georgia State, and has a very big website on his own domain. I looked at how he had arranged the teaching part of his website, and I saw that he had used a subdomain classes for all the classes he teaches, and had separate subdomains within that for each instance of each class. I didn’t want to go quite that far, but the idea of mapping each small website to a subdomain seemed like a good idea, and I could then have one website per class and re-render only the one I was working on. It took some back and forth with the person behind grebedoc.dev, but I now have that working (for example, this blog is one such small website), and I will talk about how I got it working.

One website at the root of your domain

Let’s suppose you own mydomain.com (adapt the below to your situation) and you have a Quarto website that you would like to appear when somebody visits mydomain.com. This could be (as discussed above) one big website with things in subfolders, or it could be a small one with links to all your other websites and, perhaps, your “About Me” page.

There are two things to consider:

• getting the DNS records¹ for your domain to point to the right place (there are about three parts to this)
• setting up a “webhook” in Codeberg to notify the server whenever you push an update to your website.

Your domain has a domain registrar (the people you pay money to each year to keep your domain yours). Mine is namecheap.com; godaddy.com is another one. When you log into your domain registrar’s website, you’ll see a list of the domains you have registered with them. I own two domains, so mine looks like this:

When a visitor to your domain types mydomain.com into their web browser, their browser needs to know what to show them in response. This is accomplished by converting the address the user typed into an IP address. grebedoc.dev is going to do the work of making your website visible to the world, so it is Grebedoc’s IP address that you need to provide. There are actually two forms of IP address: IPv4, the four numbers separated by dots that you are probably familiar with, and the newer IPv6, which looks like hex codes separated by colons.

To tell your domain registrar which IP address(es) it should map your domain to, you need to find the DNS settings for your domain. On namecheap.com, I click the Manage button next to the domain name, and then Advanced DNS.

A DNS “record” (probably listed as a row in the table you are looking at) has three parts:

• the Type
• the Host
• the Value

To set the IPv4 address, you need an A record. Find out how you add a new record (mine has a button under the table); there might be a dropdown where you can select the Type of record you want to add. For your A record, set

• Type as A
• Host as @
• Value as 185.187.152.7

The @ means, loosely, “the root of my website”. The Value given here is, at this writing, the IPv4 address of grebedoc.dev.

Now we set the IPv6 address. This needs an AAAA record:

• Type as AAAA
• Host as @
• Value as 2a05:b0c4:4:1::3

This last is the IPv6 address of grebedoc.dev (at this writing).

There is a third thing to add, which is how Grebedoc knows that you actually control the domain (otherwise, anyone could put up a website on your domain, or take down the one you put up there). In the Grebedoc docs, this is called Method A. For this, you need:

• Type as TXT
• Host as _git-pages-repository
• Value as https://codeberg.org/username/reponame.git

The last one is the Git HTTPS clone URL of the Codeberg repo where your rendered website lives. Substitute appropriately for your situation. Be careful to get the right Host; there should be an underscore at the front, and minuses between each word. Here are mine, after I am finished (the Host for the TXT record only shows part of the value):²

Save all of these changes.

Changes to the DNS settings can take a few minutes to propagate. If you want to find out when this has happened, you can use the dig tool. When everything is working, you should see something equivalent to this:

dig requires a domain name, possibly with a subdomain on the front, and the Type of DNS record you want to look up. You should see an ANSWER SECTION in the output. For the A record, this should have your domain name first, and the Grebedoc IP address at the end. For the TXT record, the ANSWER SECTION should have the HTTPS git clone address of your Codeberg repo.

Before the propagation has completed, there will be either no answer section, or it will show something else.

There is one more thing to do, and that is to set things up so that whenever you push your website, the server is notified and the website your users see is updated. This is done by using a Webhook in Codeberg (analogous to a Github Action). Go back to your repo in Codeberg. Click on Settings, and then Webhooks, and then Add Webhook, and then Forgejo. There are two things to change: the Target URL at the top, which should be the http version³ of your domain, as in http://mydomain.com/ with a trailing slash. Go down to Branch filter, and change that to pages (because you are using the pages branch, and when that changes, the server needs to be notified). Click Update webhook.

Now, the next time you push your website, it should appear at your domain. It can take a few minutes for everything to come together. If you need to troubleshoot, make sure the dig output above makes sense, and make sure the webhook is working properly. To do that, go to Settings and Webhooks again. This time, you should see your domain listed among the webhooks. Click on the pencil on the right. At the bottom, you should see one or more Recent Deliveries (one for each push):

If all went well, you should see a green checkmark next to the most recent of these, and clicking on the blue commit ID will give you more detail. You want the Response code to be a green 200 (“all good”), or at least one of the other 200 codes. I sometimes get a 202 response, which means that it took longer than expected, but it is working. This one is A-OK:

And now, with a little luck, you can visit your domain and see your website!

Using subdomains

I mentioned earlier that you can have one big website or lots of little ones. To do the latter, you map each little website to a subdomain of your domain. For example, the website for the course I teach starting next week is at http://stad29.ritsokiguess.site.

The process for doing this is a lot like the one for setting up a website at the root of your domain:

• setting two DNS records
• setting up a webhook in the Codeberg repository

As before, changes to the DNS records will take time to propagate, so that is the thing to do first. This time, we need a CNAME record and another TXT record:

• Type as CNAME
• Host: the name of your subdomain (only), such as blog
• Value as grebedoc.dev.

The Value needs to have a dot on the end. Then:

• Type as TXT
• Host as _git-pages-repository followed by a dot followed by the name of your subdomain (only), for example _git-pages-repository.blog
• Value as https://codeberg.org/username/reponame.git: the repo containing the website you want to map to the subdomain (in my case, https://codeberg.org/nxskok/blog.git)

Save these. You can use dig to check on progress:

and (noting the input for the second one)

The final step is the webhook. This is the same as before, except that you include the subdomain. This is the one for my blog:

Push to this repo, maybe wait a few minutes, and then check to see what the website at your subdomain looks like.

Reminder

If you have many little websites, the logical way to link them together is to have your “main” site (the one at the root of your domain) have links to all the subdomain sites, and for the subdomain sites to have a link back to the main one. Bear in mind, though, that these are independent sites, so these links should be actual URLs rather than local links.

On your main site, you might do this with a navigation bar, a navbar in Quarto jargon. My main site has a _quarto.yml file that looks like this:

project:
  type: website
  output-dir: .

website:
  title: "Ken's website"
  navbar:
    left:
      - href: index.qmd
        text: Home
      - href: http://programs-courses.ritsokiguess.site/
        text: Programs and courses
      - href: http://stac32.ritsokiguess.site/
        text: STAC32
      - href: http://stac33.ritsokiguess.site/
        text: STAC33
      - href: http://stad29.ritsokiguess.site/
        text: STAD29
      - href: http://datafiles.ritsokiguess.site/
        text: Data files
      - href: http://pasias.ritsokiguess.site/
        text: PASIAS
      - href: http://lecture-notes.ritsokiguess.site/
        text: Lecture notes
      - href: http://blog.ritsokiguess.site/
        text: Blog
      - text: "Quercus"
        href: http://q.utoronto.ca

  sidebar:
    contents: auto

format:
  html:
    theme: cosmo
    css: styles.css
    toc: true

Each of the entries in the navbar is a link to one of the subdomain websites, so the href part of the link is an actual URL. (If you have one big website, these links would be to files in the same folder system, but in my situation that is no longer the case.)

Credits

Photo credit: Bengt Nyman on Wikimedia Commons. It is a Great Crested Grebe; “grebedoc” is Codeberg backwards.)

Thanks to Catherine “Whitequark” for putting grebedoc.dev together.

Introduction

It is New Year’s Eve, and it is snowing here in Toronto, so I would rather be inside writing this.

One of the things Quarto can produce is a static website (meaning, it turns your content into a website that a user can look at but not interact with). This is ideal for websites for courses or workshops, where the idea is to provide material for your students to work with, or (at least in part) for a blog, where the idea is to share your ideas with the world. (In a blog, you might want to enable comments, but that is another story that we do not get into here.) A standard way to share these with the world was to use Github Pages, where you set your website up as a Github repo, and then configured the Pages part of Settings, and got a URL that would display the site.

There are, these days, good reasons not to use Github. There are other places to host your code that are not owned by big corporations (Microsoft, in the case of Github) and are not in the US. Some of these are based on Forgejo, such as codeberg.org, hosted in Germany, and worktree.ca, hosted in Canada. I use both of these two. You can also host Forgejo yourself. These all still use Git, so you can handle a lot of things the same way you are used to (commit, push, pull, etc).

One thing that Github Pages has going for it is that the mechanism is fairly simple, and Quarto has a “publish gh-pages” that will handle most of the details for you. Codeberg has its own equivalent, Codeberg Pages, that works a bit differently. The website you want to publish has to be two things:

• on a branch called pages
• in the root folder of the repo.

I have a couple of ways of making this work, which may not be elegant, but they appear to work. The elegance, I leave to others.

In part 2, I discuss how you can use your own domain with all of this.

Website and source in the same place

This is about the least aesthetic way to do it: have your .qmd files and the .html files produced from them in the same folder. This means that when you open up your website project in R Studio or wherever, you will see your .html files mixed in amongst your source. If you are prepared to put up with this, I think this is the easiest way to go.

When you create a Quarto website, you get a file called _quarto.yml in which lives the design of your website. Here is the top of one of mine:

The key thing is the output-dir: . — this means to build the website in the same folder as your .qmd files with your content. This solves the second Codeberg Pages bullet point above: it doesn’t matter that there are other files in with the .html ones, because the .html files are the ones that will be shown on your website.

The other thing is that first bullet point: “on a branch called pages”. Branches in git used to scare me, because the discussion of them you would find online tended also to get mixed up with “pull requests” and dealing with other people working on a project, neither of which are likely to apply to you working solo on a website. I discovered that you can have just one branch called pages and work on it all the time.

If you have used Happy Git with R, the mechanism you start with on Codeberg is like the one in “existing project, Github first” (Chapter 16 there). On Codeberg, you have to create a repo there first, and only then put stuff in it, even if you have a project in R Studio that maybe already has been attached to git.

So, log into Codeberg, and create a new repository by clicking on the + sign top right and then New Repository. When you do that and give it a name, you’ll be greeted by this:¹

These are your instructions for attaching your R Studio website to Codeberg. I find the SSH approach easier, so this is that (see Happy Git with R, chapters 9 and 10, for discussion of SSH vs HTTPS). Click the button to see the other one. However, we are going to set up a pages branch right away, so what you need to do is replace each instance of main in these instructions with pages. If your website is already a (local) git repository, add git switch -c pages before the two lines shown (and, if you like, get rid of your other branch which is probably called main). Remember that the last thing you do is git push -u origin pages, which will connect the local branch pages that you just created with the remote branch of the same name on Codeberg.

So now I have a local project, which I called disposable1, with (more or less) the template website in it, which I rendered, and then the whole thing was pushed to Codeberg.

The final step is to use Codeberg pages to share the website with the world. The documentation for Codeberg Pages is here. The key URL format is this one: https://username.codeberg.page/reponame/, so now I should be able to point my browser at https://nxskok.codeberg.page/disposable/, and this is what I see:

There are no special Settings requirements for this: any Codeberg repository with a pages branch and an index.html file in the root directory can be accessed in this way.

Website and source in different places

The approach described above works² for most of my websites, but when I came to reorganize my blog as a Quarto blog website, it didn’t seem to like having the content and the html files in the same place. This is possibly because a Quarto blog has a structure that expects certain files to be in certain places, and it doesn’t like you tinkering with it. When you render a blog, or any other Quarto website that does not have an output-dir: line in _quarto.yml, the rendered site ends up in its entirety in a folder _site below the root of your R Studio project.

You’ll remember that for Codeberg Pages to work, the website has to be in the root folder of the repo, and now it is not. So we need a different solution. The simplest one I could come up with is to maintain two repos, one with the source code and the website in the _site folder, and the other with just the contents of the _site folder, with an index.html at the root.

Using my blog as an example, the two repos are called blog1 with the source .qmd files, and blog with the rendered website.

blog1 is a Quarto website project, in the same way that disposable1 was. This, however, is not going to be shared with the world, so you can work on the main branch as usual, and it could be a private repo or not on Codeberg (I keep blog1 on worktree.ca). Create this repo wherever you are going to keep it, and follow the instructions to connect it to your local repo, as written (with the main branch). The other thing to say about this is that the _site folder that will be created each time you render your website does not need to be under version control (it can always be rebuilt), so the _site folder should be in .gitignore for this repository.

The local version of the second repository should be a subfolder in the same containing folder that blog1 is a subfolder of. (I have a folder called r-projects that all my R projects live in, so that both blog1 and blog are subfolders of r-projects.) This is not an R project of any sort, so we will use the command line to work with it. I will be using bash; adjust as necessary to your setup. To start with, blog should be an empty folder.

Then fire up a command line (terminal), and put yourself in the blog folder. (As a check, ls should return no files.) Make sure that you rendered the website blog1, so that blog1 has a subfolder called _site. Then copy the files in _site into blog, like this:

cp -R ../blog1/_site/* .

The -R recursively copies any subfolders of _site. The * on the end of the copy command is important; if you omit this, blog will contain a subfolder called _site, with the website below that, and this is not what you want. To check that it worked, run ls now, and you should see a file called index.html, along with some other files and possibly subfolders. (If it’s a blog, you’ll also see about.html and a subfolder called posts.)

The contents of blog now need to be put under version control, in a branch called pages, something like this:

git init
git switch -c pages
git add .
git commit -m "first commit"

The advantage to running git switch first is that no other branches get created, not even main.

Next, we connect a new Codeberg repo with this one. In Codeberg, create a new repo (mine has the same name blog as the local one), and follow the instructions for pushing an existing repository, remembering to use the pages branch on both ends, something like this:

git remote add origin ssh://git@codeberg.org/username/blog.git
git push -u origin pages

Now, you can check out the current state of your website at https://username.codeberg.page/reponame/, substituting your username (mine is nxskok) and repo name (blog).

From here on, the workflow is this:

• add some content to your website, in blog1
• run quarto render to render the site (creating a new version of _site inside blog1)
• go to your terminal and run the cp command (above) to copy the new version of your website into blog
• also run the git commands to push the new version of your website to Codeberg
• see how the new version of the published website looks
• rinse and repeat. (From time to time, also commit the source project blog1.)

I found it useful to create a shell script to do the copying and the git commands, since they are the same every time. Mine is called copy.sh. It lives in blog and looks like this:

cp -R ../blog1/_site/* .
git add .
git commit -m "update site"
git push

This way, I only need to run sh copy.sh to copy the new version of the website into blog and send it to Codeberg. My commit messages are very uninformative, but that hasn’t tripped me up yet.

Looking ahead

In part 2, I talk about using your own domain with Codeberg.

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(tidyLP)

Introduction

Yesterday on Mastodon I saw this post about the tidyLP package. I was thinking “I have a spiffy new blog (with the same un-spiffy old posts) and I haven’t written anything in 2025, so maybe I should write about my experiences with the package before the year is out”.

A back-story: I moved to the UK to teach for a bit (this was some years ago) and learned that Operations Research (linear programming, integer programming, transportation and assignment problems, etc.) fell under the remit of Statistics, even though there is no actual randomness in any of the above, so I was expected to teach that stuff even though I had not used it in a long time, or ever. So I had some late nights learning or re-learning material to teach to my students in the next week. The book we used was the massive tome by Winston (this edition). The link doesn’t say how many pages, but “weight: 2.45 kg” gives you the idea. Why we were using a massively wordy American text rather than a much more concise British one, I have no idea.

A first example

One of the first linear programming problems in Winston goes like¹ this:

A toy company makes wooden toy soldiers and toy trains. Each soldier requires 2 hours of finishing time and one hour of carpentry time, and results in a profit of $3. Each train requires 1 hour of finishing time and 1 hour of carpentry time, and results in a profit of $2. There are 100 finishing hours and 80 carpentry hours available, and previous experience indicates that a maximum of 40 soldiers can be sold. How many soldiers and trains should be made in order to maximize profit?

There are two things that make this a linear programming problem:

• the function to be optimized (here profit) is linear
• there are some (also linear) constraints on what we can do.

The standard by-hand method of solving these problems is called the Simplex Method. The idea is that the constraints define a region in space called a simplex, and the highest profit is at one of the corners of the the simplex. The Simplex Method tells you how to move from corner to corner while increasing the profit (and thus, when there is nowhere else to move to, you have found the optimal solution).

Unless you are learning the method specifically, these days there is no need to learn the Simplex Method to solve these problems, and the tidyLP package provides a function lp_solve to find the solution for you.

In this problem, the logical flow is:

• the number of soldiers and trains you make determine how much profit you make.
• the number of soldiers and trains made also determine how many finishing and carpentry hours you need
• there are limits on all of those things, which you need to respect.

The way you set this up in tidyLP is to begin with a dataframe which captures the relationships in the first of those two bullet points. Each row of that dataframe contains something you want to optimize over, and each column contains the contribution of each additional unit of those things to your profit or to your finishing and carpentry hours, like this:²

d_csv <- "
product, finishing, carpentry, profit
soldier, 2, 1, 3
train, 1, 1, 2
"
d <- read_csv(d_csv)
d

This takes care of the first part of the problem statement.

Our constraints come in two flavours:

• ones that are related to columns of the dataframe we just made (the ones on finishing and carpentry hours)
• ones that are related to the variables we are optimizing over (in this case, the limit on the number of soldiers made).

These go into the problem specification in different ways.

Using tidyLP to solve this problem goes in three steps, which can be “piped” together (hence “tidy”):

• use tidy_lp to specify the problem and constraints
• use lp_solve to solve the problem
• use bind_solution to glue the solution onto our dataframe

I then like to save the latter, and then inspect it. Here’s how it goes for this problem:

tidy_lp(
  d,
  profit,
  finishing ~ leq(100),
  carpentry ~ leq(80),
  (product == "soldier") ~ leq(40)
) %>% 
  lp_solve() %>% 
  bind_solution() -> sol
sol %>% select(product, .solution)

The inputs to tidy_lp are our dataframe, the column of it to optimize (by default maximize), and then one for each constraint. For constraints expressed in terms of the columns of our dataframe, you put the column name on the left of a squiggle, and on the right you put for example leq (“less than or equal to”) with a number inside saying what it is less than or equal to. For constraints expressed in terms of what we are optimizing over (the soldiers constraint here), the notation is different: the above shows how we express “the number of soldiers is less than 40”. I put soldiers and trains in a column called product.

For constraints, there are also geq() and eq(), which are “greater or equal” and “exactly equal” respectively.

The last step shows that the optimal solution is to make 20 soldiers and 60 trains. This happens to be the same answer as Winston got, so it is presumably correct.

We can go further, and investigate how many of our finishing and carpentry hours we ended up using. For this, we note that what I called sol has all the columns in our original dataframe in it, so we can do calculations like this:

sol %>% mutate(across(finishing:profit, \(x) x * .solution))

sol %>% summarize(across(finishing:profit, \(x) sum(x * .solution)))

This shows that we made a profit of $60 from soldiers and $120 from trains for a total of $180. Also, we used 40 finishing hours for soldiers and 60 for trains, so we used up our total 100 finishing hours. For carpentry hours, we used 20 for making soldiers and 60 for making trains, so we also used up our total of 80 of these. We only made 20 soldiers, though, not reaching our limit of 40: this constraint is not “binding”.

None of this, of course, is actually a proof that the answer is best (other than our confidence in lp_solve): it respects all the constraints, but in principle there could be a better number of soldiers and trains to make that also respects the constraints.

A second example

This one comes directly from Winston (Problem 3 in Section 3.2 in my edition):

Leary Chemical manufactures three chemicals: A, B, and C. These chemicals are produced via two production processes: 1 and 2. Running process 1 for an hour costs $4 and yields 3 units of A, 1 of B, and 1 of C. Running process 2 for an hour costs $1 and produces 1 unit of A and 1 of B. To meet customer demands, at least 10 units of A, 5 of B, and 3 of C must be produced daily. Graphically determine a daily production plan that minimizes the cost of meeting Leary Chemical’s daily demands.

Note that in this problem, there is no benefit to producing extra of the chemicals beyond what is needed to meet customer demands: here, we are not selling it at a profit, but producing it at a cost.

The logical flow is that the number of hours processes 1 and 2 are run (to be optimized over) determines the amounts of chemicals A, B, and C that are produced, so these are the relationships that need to be summarized in our dataframe. I have called the processes P1 and P2 (so that I don’t get confused):

d_csv <- "
process, A, B, C, cost
P1, 3, 1, 1, 4
P2, 1, 1, 0, 1
"
d <- read_csv(d_csv)
d

Process 2 does not produce any of chemical C, hence the zero in the dataframe.

The constraints (meeting customer demand) are all in terms of columns of this dataframe (there are, for example, no limits on how many hours we can run each process for), so there is no difficulty in expressing the constraints:

tidy_lp(
  d, 
  cost, 
  A ~ geq(10),
  B ~ geq(5),
  C ~ geq(3),
  .direction = "min"
) %>% 
  lp_solve() %>% 
  bind_solution() -> sol
sol %>% select(process, .solution)

The one new thing is that this is a minimum-cost problem. This is specified by using the additional input .direction.

The solution is to run process 1 for 3 hours and process 2 for 2 hours. How much of each chemical is produced under this plan?

sol %>% mutate(across(A:cost, \(x) x * .solution))

sol %>% summarize(across(A:cost, \(x) sum(x * .solution)))

There is no extra of chemicals B and C produced, but this plan produces one extra unit of chemical A. This seems to be unavoidable: to produce enough of C, process 1 has to be run for 3 hours, and to produce enough of B, process 2 has to be run in addition for 2 hours. Doing that produces units of A.

Being a statistician

As I said earlier, this is a mathematical problem and not a statistical one. Suppose we had no knowledge of the simplex method or the tidyLP package. What would we do? One way is to generate some numbers of hours to run each process, work out the cost of doing so and the amounts of the three chemicals produced. This part is a bit complicated, so we’ll put it in a function, imaginatively called f:

f <- function(d, p1, p2) {
  v <- c(p1, p2)
  d %>% summarize(across(A:cost, \(x) sum(x * v)))
}

This returns a one-row dataframe with the total cost and the amounts of A, B, and C produced:

f(d, 3, 2)

reproducing our answer from earlier.

All right, let’s generate some literally random³ number of hours to run each process (let’s say between 0 and 5):

nrow <- 7
max_hours <- 5
g <- tibble(p1 = runif(nrow, 0, max_hours), 
            p2 = runif(nrow, 0, max_hours))
g

Now, we can run our function for each of those. My function is not vectorized,⁴ so I use rowwise for this:

g %>% 
  rowwise() %>% 
  mutate(ans = list(f(d, p1, p2)))

and now I unnest ans wider to see the actual values:

g %>% 
  rowwise() %>% 
  mutate(ans = list(f(d, p1, p2))) %>% 
  unnest_wider(ans)

I cannot just minimize the cost here; I need to make sure I produced enough of each chemical:

g %>% 
  rowwise() %>% 
  mutate(ans = list(f(d, p1, p2))) %>% 
  unnest_wider(ans) %>% 
  mutate(constr_ok = (A >= 10 & B >= 5 & C >= 3))

Only two of these rows satisfy all the constraints, so I filter those and find the one with the smallest cost:

g %>% 
  rowwise() %>% 
  mutate(ans = list(f(d, p1, p2))) %>% 
  unnest_wider(ans) %>% 
  mutate(constr_ok = (A >= 10 & B >= 5 & C >= 3)) %>% 
  filter(constr_ok) %>% 
  slice_min(cost)

Using this amazingly mindless method, our best cost is 17.9 (compared to the actual optimal 14), running process 1 for 4.10 hours and process 2 for 1.49 hours. But at least it satisfies the constraints.

The thing about simulations is that you can get an answer as close as you like by running it long enough. Let’s try 1000 simulations this time. The code is all there; this time we redefine g:

nrow <- 1000
max_hours <- 5
g <- tibble(p1 = runif(nrow, 0, max_hours), 
            p2 = runif(nrow, 0, max_hours))
g

and then run the exact same code again:

g %>% 
  rowwise() %>% 
  mutate(ans = list(f(d, p1, p2))) %>% 
  unnest_wider(ans) %>% 
  mutate(constr_ok = (A >= 10 & B >= 5 & C >= 3)) %>% 
  filter(constr_ok) %>% 
  slice_min(cost)

and this time we have gotten much closer to the answer that tidyLP gave us: a cost of 14.4 from running process 1 for 3.03 hours and process 2 for 2.30 hours, and we can check that at least enough of each chemical was produced.

Simulation for the win. Or something like that.

Integer programming

The tidyLP package can also handle “integer programming”, where the answers must be integers or a binary “yes/no”. In the package README, the author gives an actual practical non-toy example of selecting a fantasy basketball team, given limits on the total salary and requirements for players playing each position and the number that can be from the same NBA team. There are over 300 players to select from, and each player is either in or not in the fantasy team (so it is a binary problem).

Here is a rather artificial integer problem, based on problem 15 in section 9.2 of Winston:⁵

Fruit Computer makes two types of computer. A Pear computer requires 1 hour of labour and 2 chips, and sells for $400; an Apricot computer requires 2 hours of labour and 5 chips, and sells for $900. There are 23 chips and 15 hours of labour available. The company should make at least as many Apricot computers as Pear ones. How many computers of each type should Fruit Computer manufacture to maximize revenue?

The setup for this is the same as for a regular linear programming problem:

d_csv <- "
computer, labour, chips, selling
Pear, 1, 2, 400
Apricot, 2, 5, 900
"
d <- read_csv(d_csv)
d

This time, we have a constraint in terms of computers (not just the columns of the dataframe d). The right side of a constraint has to be a number, so we write this as “the number of Apricots made minus the number of Pears made must be greater or equal to zero”. With that in mind, and solving as before, we get:

tidy_lp(
  d, 
  selling, 
  labour ~ leq(15),
  chips ~ leq(23),
  (computer == "Apricot") - (computer == "Pear") ~ geq(0)
) %>% 
  lp_solve() %>% 
  bind_solution() -> sol
sol %>% select(computer, .solution)

Obviously, we cannot make any fractional computers. To force the answers to be integers, set .all_int to be TRUE:

tidy_lp(
  d, 
  selling, 
  labour ~ leq(15),
  chips ~ leq(23),
  (computer == "Apricot") - (computer == "Pear") ~ geq(0),
  .all_int = TRUE
) %>% 
  lp_solve() %>% 
  bind_solution() -> sol
sol %>% select(computer, .solution)

The best solution is to make 1 Pear computer and 4 Apricot. Note that this solution is not nearly the same as the decimal solution to this problem rounded off. Making 3 Pears and 3 Apricots uses 9 hours of labour and 21 chips, so it satisfies the constraints, with a revenue of $3900, but…

sol %>% 
  mutate(across(labour:selling, \(x) x * .solution))

sol %>% 
  summarize(across(labour:selling, \(x) sum(x * .solution)))

The revenue is $4000, more than from making 3 computers of each type. This solution uses 22 of the 23 chips and only 9 of the 15 available hours of labour. Requiring the answers to be integers has limited us a good bit.

Two scenarios

The first scenario is based on the infamous mtcars dataset. I put this in the file motor-trend.qmd, which looks like this:

## Motor Trend cars

In 1974, the *Motor Trend* magazine collected data on fuel consumption
and other features of 32 different makes of car. The data are available
in the built-in dataset `mtcars`. The variables of interest to us are:

- `mpg`: fuel consumption in miles per US gallon
- `cyl`: number of cylinders in the engine
- `wt`: weight of car, in thousands of pounds.

(@) Make a suitable plot of fuel consumption against weight.

(@) Modify your plot to distinguish cars with different numbers of cylinders by colour.

This is not a self-contained Quarto file: it’s Quarto all right, but it’s designed to be included in another file (which it will be, later). In the Quarto documentation, they recommend giving a file like this a name with an underscore on the front, to make sure it doesn’t get rendered by accident (if, for example, the folder is a Quarto project and you render the whole folder). I’m going to control things with Targets, however, so I’m not going to worry about that.

The other notable feature here is how I label the two questions: the (@) on the front, which will auto-number them from 1 upwards in the final worksheet.¹

The second scenario is based on some data on making soap, which lives in soapy.qmd:

## Making soap

A factory makes soap. There are two production lines, `a` and `b`. 
These can be run at different speeds; running the production line faster
produces more soap, but it also produces more scrap (soap that cannot be
sold). Does the amount of scrap differ by production line? Answer the
questions below to find out. The data is in
<https://ritsokiguess.site/datafiles/soap.txt>.

(@) Read in and display some of the data.

(@) Make a suitable plot of the scrap produced and the production line. How do the production lines compare?

(@) Do you get a different story if you include speed in your plot?

This is structured the same way as the first file, and will have three numbered questions when it is rendered.

Adding the solutions

This, I have to admit, I stole more or less wholesale from Nicola Rennie, whose blog post you would do well to read. There are two key ideas:

• Parameterised documents
• Conditional content

In the YAML header of a Quarto document, you can have a section called params which supplies some default values for parameters. The example in the Quarto documentation is this:

---
params:
  alpha: 0.1
  ratio: 0.1
---

which sets default values for the parameters alpha and ratio. You access them in the Quarto document through R like this, in an R code block:

params$alpha

You can supply different values by running quarto render with the -P option, like this:

quarto render myfile.qmd -P alpha:0.2 -P ratio:0.3

and then 0.2 and 0.3 will get passed down into your document.

Wait, you say, what YAML block? Neither of our files even have a YAML block. Well, when we get around to making the worksheet itself out of our two scenarios, we’ll have a proper “main” Quarto document that includes our two files, and not only will that have a YAML header with default parameter values in it, but also they will get passed down into our “child” documents with the scenarios in. The parameter we will use will be called hide_answers and will be either true or false.

All right, now to conditional content. Here’s how you hide some content if you are creating an HTML document (from the Quarto docs):

::: {.content-hidden when-format="html"}

Will not appear in HTML.

:::

The ::: marks the beginning and end of a so-called “div block”. Inside the {} on the top line is a class that the text has (being hidden) and an optional condition when it should be hidden (when the document format is HTML).

That’s all fine and wonderful, but we want to make our content hidden when something in R is true (namely, params$hide_answers is TRUE). The way around this is to use inline R code to produce the top and bottom lines of our div block:

at the top, and

at the bottom. The way this works is if params$hide_answers is TRUE, these lines create a div block with the content-hidden class (that is, the text between these two lines is hidden), but if params$hide_answers is FALSE, no div block is created at all, and the text between these two lines is displayed.

Now we have the machinery to add some optionally-displayable text, that is to say, solutions, to our problems. What you do is to add the code that optionally starts the div block at the start of a solution, and the code that optionally ends it at the end. Thus, for example, the Motor Trend question file with solutions² looks like this:

This process for adding solutions to a file of questions really ought to be called Renniefication.

Making a worksheet

Now that we have scenarios, questions and solutions, we can put together our Worksheet 99. This is how it goes together, with some comments below:

• In the YAML block at the top:
  • I include df-print: paged to make dataframes (in the solutions) display nicely, and, as a pre-emptive strike, embed_resources: true to make sure my output HTML doesn’t lose any of its graphs if the file gets moved around.
  • This is where my params block goes. I have one parameter here, the hide_answers that I mentioned earlier, which I have set to true here.
• In the diminutive body of this document, I have space for overall instructions, and loading of any packages the worksheet might need.
• The separate scenario files are loaded using the include Quarto “shortcode”. I think this is the cleanest way to do it, but you could also use R Markdown style “child documents” here. This works as if the file contents have been literally copied and pasted where the include is, and has the effect that the parameters (that is, the value of hide_answers) are passed down into the included files. When I refer to params$hide_answers inside motor-trend.qmd (as I did above), it uses the right value and correctly includes or excludes the solutions.

This is, you might say, the end of the story. You render this file once with hide_answers: true to make a worksheet to give to your students, and later you change true to false to make the solutions for them.

However, there is more human intervention here than you might like. Both the question document and the solutions document will be called worksheet_99.html, and you’ll have to remember (or look) to find out whether it currently contains questions or solutions. It would be nice to make two html files, one with just the questions and the other with solutions as well, each with different names like worksheet_99_q.html and worksheet_99_a.html.

The other thing is how to keep everything up to date. If you change either of the included files, you want to be able to re-render worksheet_99.qmd without having to remember to do so. Veteran Fortran programmers like me would solve this with a Makefile. The R way to do this is to use the targets package, which we discuss shortly.

Rendering with parameters

One way to supply parameter values is to put them in params in the YAML block. But you can also supply them on the command line if you render that way, like this:

quarto render worksheet_99.qmd -P hide_answers:true

This puts the questions without solutions into worksheet_99.html. But we can go one step further and set the name of the output file, like this:

quarto render worksheet_99.qmd -P hide_answers:true -o worksheet_99_q.html

Then, to make an HTML file with the solutions as well, you change hide_answers:true to hide_answers:false and change the -o part to -o worksheet_99_a.html.

There is enough repetitive stuff here that I wrote a function to do it:

renderify <- function(fname, ...) {
  ans <- SplitPath(fname)
  qq <- str_c(ans$filename, "_q.html")
  aa <- str_c(ans$filename, "_a.html")
  cmd <- str_c("quarto render ", ans$fullfilename)
  cmd1 <- str_c(cmd, " -P hide_answers:true -o ", qq)
  cmd2 <- str_c(cmd, " -P hide_answers:false -o ", aa)
  olddir <- setwd(ans$dirname)
  system(cmd1)
  system(cmd2)
  setwd(olddir)
  fname
}

I wrote this just before reading Danielle Navarro’s excellent blog post on the fs package, and now I realize that this would have been a great reason to learn about that package. I had, however, gotten this working using SplitPath from the DescTools package, so this is what you get. Also, I realize, now that I look at the code, that it would have benefitted greatly from using glue::glue rather than str_c from stringr.³

Girt af.

Anyway: the function takes as input a filename (of a .qmd file) and:

• splits the input filename up into folder, base filename, and extension.
• constructs two output filenames by gluing _q.html and _a.html onto the end of the base filename
• constructs the common part of the quarto render command. fullfilename is the base filename plus its extension but not including its folder. This is important for reasons we see in a moment.
• constructs the full quarto render commands using the -P and -o options we saw above.

So now I run cmd1 and cmd2 that I so laboriously constructed, right? Not so fast. When you run quarto render from the command line, the file you’re rendering has to be in the same folder that you currently are. This is not usually the case for me: my project has worksheets and assignments in a subfolder assignments, and when I am running targets that is all controlled from the main project folder. So, very carefully, I change to the subfolder the .qmd is in (saving my previous folder to go back to later), then run my commands, then go back to the folder I was in.

I hope in this way I am safe from having my computer set on fire, although I could undoubtedly stand to learn about the here package too.

Doing this in targets

The (very brief) idea behind targets is that certain of your files (like documents) depend on certain other things (included files, here, or functions or datasets in general). The Targets book has a great intro walkthrough. What you do is to create “targets”, and then have functions that express how targets depend on each other. The definition of the targets lives in a file _targets.R in the project folder. Here is the relevant bit of that:

worksheet99 <- list(
  tar_target(worksheet_99_file, "assignments/worksheet_99.qmd", format = "file"),
  tar_target(motor_trend, "assignments/motor-trend.qmd", format = "file"),
  tar_target(soapy, "assignments/soapy.qmd", format = "file"),
  tar_target(worksheet_99, renderify("assignments/worksheet_99.qmd",
                                      worksheet_99_file,
                                      motor_trend,
                                      soapy))
)

My function renderify is in a file R/functions.R which has been sourced earlier in _targets.R.

First, I create targets for each of my three files: the two question files, and the main worksheet file. Inside tar_target, the first thing is the name of the target you’re making, the second is where the file lives, and the third thing is format = "file". Then, the last target is a function call. As we have seen, renderify creates the two output files for the worksheet, but also serves the double duty of enforcing the dependency of the final worksheet on the targets made from the other three files. Targets knows this because those other three targets are also input to renderify, so if any of those three targets have changed (meaning, any of the files from which those targets are made), the whole worksheet will be re-rendered. The relevant part of targets::tar_visnetwork shows this: target worksheet_99 depends on targets worksheet_99_file, motor_trend, and soapy.

The project from which the above was taken has targets for a whole bunch of worksheets, assignments, tests etc. If you look at targets examples, you will usually see, at the end of the file, a list() that defines every single one of the targets. But for me, this was getting out of hand, so I defined and saved a separate list() for each worksheet or assignment, and then at the end of my _targets.R, I glue them all together into one list with some code like this:

c(worksheet1, worksheet2, worksheet3, worksheet4, 
  worksheet5, worksheet6, worksheet7, worksheet8, 
  worksheet9, worksheet99)

Now, I can run targets::tar_make() and my worksheet will be re-rendered (twice, to get the two output files) if and only if any of the files making it up have changed.

You might be thinking that renderify doesn’t need those other inputs, and you would be quite right: the only thing the function uses is the filename that is the first input. I used ... in my function code to allow arbitrary many other inputs, and these are used only to create the dependency, so that Targets knows what depends on what. This is the cleanest way I could think of.

Results

So maybe now you want to see the result of all of this:

• questions file
• solutions file

An example

One of the benefits of exercise is that it stresses bones and makes them stronger. Researchers at Purdue did a study in which they randomly assigned rats to one of three exercise groups (“high jumping”, “low jumping” and a control group that was not made to do any jumping). The rats were made to do 10 jumps a day, 5 days a week, for 8 weeks, and at the end of this time each rat’s bone density was measured. Did the amount of exercising affect the bone density, and if so, how?

A starting point is to read in the data and make a graph. With one quantitative and one categorical variable, the right kind of graph is a boxplot:

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

my_url <- "http://ritsokiguess.site/datafiles/jumping.txt"
rats <- read_delim(my_url," ")

Rows: 30 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: " "
chr (1): group
dbl (1): density

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

rats

# A tibble: 30 × 2
   group   density
   <chr>     <dbl>
 1 Control     611
 2 Control     621
 3 Control     614
 4 Control     593
 5 Control     593
 6 Control     653
 7 Control     600
 8 Control     554
 9 Control     603
10 Control     569
# ℹ 20 more rows

and then

ggplot(rats, aes(y=density, x=fct_inorder(group))) + geom_boxplot()

The boxplot shows that the bone density is much higher on average for the high-jumping rats than for the others; there seems to be not much difference between the control rats and the ones doing low jumping.

An annoying detail is that ggplot will put the groups in alphabetical (= nonsensical) order unless you stop it from doing so. In the data read in from the file, the jumping groups are in a sensible order, so I can use fct_inorder from forcats to arrange the group categories in the order they appear in the data.

Under the standard assumptions for analysis of variance (which we don’t assess in this post), the ANOVA -test is obtained this way:

rats.1 <- aov(density ~ group, data = rats)
summary(rats.1)

            Df Sum Sq Mean Sq F value Pr(>F)   
group        2   7434    3717   7.978 0.0019 **
Residuals   27  12579     466                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The null hypothesis, which says that all three groups have the same mean bone density, is rejected. So, not all the groups have the same mean. But that’s all we learn here. We learn nothing about which groups differ from which, which is what we really want to know.

Some math

Let’s see how far we can get with math. Let’s assume the null hypothesis is true (that all of our groups have the same mean ), and we’ll also assume that all the observations have a normal distribution with the same variance (a standard assumption of ANOVA), and, to make life easier, that all the groups have the same number of observations .

Let denote the th observation in group . Then, each under our assumptions has independently a normal distribution with mean and variance . Hence, each group’s sample mean has a normal sampling distribution with mean and variance . (Having each group be the same size makes these variances the same.)

Tukey’s idea was: one of the sample means will be largest, just by chance, and one of them will be smallest, just by chance. What is the distribution of the largest one minus the smallest one? This leans on a known result: if are independently normal with the same distribution, the distribution of the largest one minus the smallest one, scaled by an estimate of spread, has what is called a studentized range distribution, for which (as we used to say in the old days) tables are available: has a studentized range distribution, which depends on and the degrees of freedom in .

Now, since the are also normal, it follows that also has a studentized range distribution, where in this case is the square root of a pooled estimate of variance, which is just the average of the within-group variances (because the sample size within each group is the same). To make this work, Tukey said that if you take the upper 5% point of this distribution, and scale it properly, you can say that any group means will rarely differ by this much if the null hypothesis is true, and thus that any two group means that do differ by more than this are significantly different.

The value of doing this is that you are only doing one test, based on how far apart the largest and smallest sample means might be, and applying that to all pairs of groups, so that you avoid the multiple testing problem of doing all possible two-sample -tests.

So, say we have three groups with 10 observations in each (as in our jumping rats data). The upper 5% point of the appropriate Studentized range distribution is

q <- qtukey(0.95, nmeans = 3, df = 27)
q

[1] 3.506426

then we multiply that by the square root of the error mean square divided by its df, with an extra factor of 2 that I am not sure about:

w <- q * sqrt(466/27*2)
w

[1] 20.60112

and we say that any sample means differing by more than that are significantly different:

rats %>% 
  group_by(group) %>% 
  summarize(mean_density = mean(density))

# A tibble: 3 × 2
  group    mean_density
  <chr>           <dbl>
1 Control          601.
2 Highjump         639.
3 Lowjump          612.

Control and Lowjump are not far enough apart to be significantly different, but the two comparisons with Highjump are both significant.

In practice, we would use TukeyHSD which does all of that for us:

TukeyHSD(rats.1)

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = density ~ group, data = rats)

$group
                  diff       lwr       upr     p adj
Highjump-Control  37.6  13.66604 61.533957 0.0016388
Lowjump-Control   11.4 -12.53396 35.333957 0.4744032
Lowjump-Highjump -26.2 -50.13396 -2.266043 0.0297843

with the same results.

Some simulation

Can we simulate the distribution of the difference between the highest and lowest sample means of our three groups? We have to set this up so that the null hypothesis is true, so that all three simulations come from groups with the same mean. It doesn’t matter what mean we use, but we may as well use the overall mean of our data:

rats %>% 
  summarize(grand_mean = mean(density)) %>% as.numeric() -> grand_mean
grand_mean

[1] 617.4333

gp_sd <- sqrt(466)
gp_sd

[1] 21.58703

so we simulate three groups of 10 observations from a normal distribution with mean 617.4333 and SD 21.5870, and then compare the highest mean with the lowest one. I’m putting this into a function because I’m going to build a simulation around it. The steps are:

• set up for as many samples as I want
• work rowwise
• draw a random sample of the right size with the right mean and SD in each row
• work out the mean of each sample
• stop working rowwise
• find the largest and smallest of the simulated sample means, and the difference between them
• return that difference as a number:

ksample <- function(nobs, nsample, mu, sigma) {
  tibble(sample = 1:nsample) %>% 
    rowwise() %>% 
    mutate(my_sample = list(rnorm(nobs, mu, sigma))) %>% 
    mutate(my_mean = mean(my_sample)) %>% 
    ungroup() %>% 
    summarize(mn = min(my_mean), mx = max(my_mean)) %>% 
    mutate(rnge = mx - mn) %>% 
    pull(rnge)
}

Does it work?

ksample(10, 3, grand_mean, gp_sd)

[1] 16.44373

Well, I guess that looks all right. So now let’s do this many times:

set.seed(457299)

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(rnge = ksample(10, 3, grand_mean, gp_sd)) -> ranges
ranges

# A tibble: 1,000 × 2
# Rowwise: 
     sim  rnge
   <int> <dbl>
 1     1  7.73
 2     2 25.1 
 3     3  3.45
 4     4 13.2 
 5     5 17.5 
 6     6 22.9 
 7     7 12.6 
 8     8 12.9 
 9     9 12.0 
10    10  2.92
# ℹ 990 more rows

The differences between the highest sample mean and the lowest one seem all over the place, but so it is:

ggplot(ranges, aes(x = rnge)) + geom_histogram(bins = 10)

Skewed to the right, with a lower limit of zero.

So now, let’s compare this null distribution with our actual data:

rats %>% 
  group_by(group) %>% 
  summarize(mean_density = mean(density))

# A tibble: 3 × 2
  group    mean_density
  <chr>           <dbl>
1 Control          601.
2 Highjump         639.
3 Lowjump          612.

We can get a Tukey-style P-value by taking each difference in means, and seeing how many of our simulated max mean minus min mean exceed that:

• control vs highjump, difference is 37.6:

ranges %>% count(rnge >= 37.6)

# A tibble: 1 × 2
# Rowwise: 
  `rnge >= 37.6`     n
  <lgl>          <int>
1 FALSE           1000

• control vs lowjump, difference is 11.4:

ranges %>% count(rnge >= 11.4)

# A tibble: 2 × 2
# Rowwise: 
  `rnge >= 11.4`     n
  <lgl>          <int>
1 FALSE            544
2 TRUE             456

highjump vs lowjump, difference is 26.2:

ranges %>% count(rnge >= 26.2)

# A tibble: 2 × 2
# Rowwise: 
  `rnge >= 26.2`     n
  <lgl>          <int>
1 FALSE            985
2 TRUE              15

With P-values

TukeyHSD(rats.1)

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = density ~ group, data = rats)

$group
                  diff       lwr       upr     p adj
Highjump-Control  37.6  13.66604 61.533957 0.0016388
Lowjump-Control   11.4 -12.53396 35.333957 0.4744032
Lowjump-Highjump -26.2 -50.13396 -2.266043 0.0297843

Our simulated P-values were respectively 0, 0.456, and 0.015, which are very much consistent with the actual ones from Tukey’s procedure.

If you wanted to do this the way we used to do it, rather than getting P-values, you get a critical value as the 95th percentile of the simulated mean differences:

ranges %>% 
  ungroup() %>% 
  summarize(q = quantile(rnge, 0.95))

# A tibble: 1 × 1
      q
  <dbl>
1  22.5

and then you’d say that any means that differed by more than 22.5 were significantly different (the two comparisons involving Highjump) and any differing by less than that were not (Lowjump vs Control).

Packages

library(tidyverse) # for purrr, the magrittr pipe, and crossing from tidyr

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Square roots (and logs) with map

sqrt(1:10)

 [1] 1.000000 1.414214 1.732051 2.000000 2.236068 2.449490 2.645751 2.828427
 [9] 3.000000 3.162278

sqrt1 <- function(x) sqrt(x[1])
sqrt1(1:10)

[1] 1

1:10 %>% map_dbl(sqrt1)

 [1] 1.000000 1.414214 1.732051 2.000000 2.236068 2.449490 2.645751 2.828427
 [9] 3.000000 3.162278

1:10 %>% map_dbl(log, 10)

 [1] 0.0000000 0.3010300 0.4771213 0.6020600 0.6989700 0.7781513 0.8450980
 [8] 0.9030900 0.9542425 1.0000000

log_base <- function(x) log(10, x)
base <- c(2, exp(1), 10) # the second one is e
base %>% map_dbl(log_base)

[1] 3.321928 2.302585 1.000000

sapply(1:10, log, 10)

 [1] 0.0000000 0.3010300 0.4771213 0.6020600 0.6989700 0.7781513 0.8450980
 [8] 0.9030900 0.9542425 1.0000000

1:10 %>% map_dbl(~sqrt1(.))

 [1] 1.000000 1.414214 1.732051 2.000000 2.236068 2.449490 2.645751 2.828427
 [9] 3.000000 3.162278

tibble(x = 1:10) %>% 
  mutate(root = map_dbl(x, ~sqrt1(.)))

# A tibble: 10 × 2
       x  root
   <int> <dbl>
 1     1  1   
 2     2  1.41
 3     3  1.73
 4     4  2   
 5     5  2.24
 6     6  2.45
 7     7  2.65
 8     8  2.83
 9     9  3   
10    10  3.16

1:10 %>% map_dbl(~log(., 10))

 [1] 0.0000000 0.3010300 0.4771213 0.6020600 0.6989700 0.7781513 0.8450980
 [8] 0.9030900 0.9542425 1.0000000

base

[1]  2.000000  2.718282 10.000000

base %>% map_dbl(~log(10, .))

[1] 3.321928 2.302585 1.000000

1:10 %>% map_dbl(\(x) sqrt1(x))

 [1] 1.000000 1.414214 1.732051 2.000000 2.236068 2.449490 2.645751 2.828427
 [9] 3.000000 3.162278

number <- 1:10
map_dbl(number, \(number) sqrt1(number))

 [1] 1.000000 1.414214 1.732051 2.000000 2.236068 2.449490 2.645751 2.828427
 [9] 3.000000 3.162278

tibble(x = 1:10) %>% 
  mutate(root = map_dbl(x, \(x) sqrt1(x)))

# A tibble: 10 × 2
       x  root
   <int> <dbl>
 1     1  1   
 2     2  1.41
 3     3  1.73
 4     4  2   
 5     5  2.24
 6     6  2.45
 7     7  2.65
 8     8  2.83
 9     9  3   
10    10  3.16

x <- 2:4
base

[1]  2.000000  2.718282 10.000000

crossing(x, base) %>% 
  mutate(log_of = map2_dbl(x, base, \(x, base) log(x, base)))

# A tibble: 9 × 3
      x  base log_of
  <int> <dbl>  <dbl>
1     2  2     1    
2     2  2.72  0.693
3     2 10     0.301
4     3  2     1.58 
5     3  2.72  1.10 
6     3 10     0.477
7     4  2     2    
8     4  2.72  1.39 
9     4 10     0.602

u <- 1:5
v <- 11:15
map2_dbl(u, v, \(u, v) sqrt1(u+v))

[1] 3.464102 3.741657 4.000000 4.242641 4.472136

Collatz

When I am teaching this stuff, I say that if the thing you are working out is complicated, write a function to do that first, and then worry about for-eaching it. For example, imagine you want a function that takes an integer as input, and the output is:

• if the input is even, half the input
• if the input is odd, three times the input plus one

This is a bit long to put in the anonymous function of a map, so we’ll define a function hotpo to do it first:⁵

hotpo <- function(x) {
  stopifnot(x == round(x)) # error out if input is not an integer
  if (x %% 2 == 0) {
    ans <- x %/% 2
  } else {
    ans <- 3 * x + 1
  }
  ans
}
hotpo(4)

[1] 2

hotpo(3)

[1] 10

hotpo(5.6)

Error in hotpo(5.6): x == round(x) is not TRUE

So now, we can use a map to work out hotpo of each of the numbers 1 through 6:

first <- 1:6
map_int(first, hotpo)

[1]  4  1 10  2 16  3

or

map_int(first, ~hotpo(.))

[1]  4  1 10  2 16  3

or

map_int(first, \(first) hotpo(first))

[1]  4  1 10  2 16  3

where we call our function in the anonymous function. The answer is the same any of these ways, and you can reasonably argue that the last one is the clearest because the inputs to the map_int and the function have the same name.

This one is map_int because hotpo returns an integer.

This function is actually more than a random function defined on integers; it is part of an open problem in number theory called the Collatz conjecture. The idea is if you do this:

10

[1] 10

hotpo(10)

[1] 5

hotpo(hotpo(10))

[1] 16

hotpo(hotpo(hotpo(10)))

[1] 8

hotpo(hotpo(hotpo(hotpo(10))))

[1] 4

hotpo(hotpo(hotpo(hotpo(hotpo(10)))))

[1] 2

hotpo(hotpo(hotpo(hotpo(hotpo(hotpo(10))))))

[1] 1

you obtain a sequence of integers. If you ever get to 1, you’ll go back to 4, 2, 1, and loop forever, so we’ll say the sequence ends if it gets to 1. The Collatz conjecture says that, no matter where you start, you will always get to 1.⁶

Let’s assume that we are going to get to 1, and write a function to generate the whole sequence. The two key ingredients are: the hotpo function we wrote, and a while loop to keep going until we do get to 1:

hotpo_seq <- function(x) {
  ans <- x
  while(x != 1) {
    x <- hotpo(x)
    ans <- c(ans, x)
  }
  ans
}

and test it:

hotpo_seq(10)

[1] 10  5 16  8  4  2  1

the same short ride that we had above, and a rather longer one:

hotpo_seq(27)

  [1]   27   82   41  124   62   31   94   47  142   71  214  107  322  161  484
 [16]  242  121  364  182   91  274  137  412  206  103  310  155  466  233  700
 [31]  350  175  526  263  790  395 1186  593 1780  890  445 1336  668  334  167
 [46]  502  251  754  377 1132  566  283  850  425 1276  638  319  958  479 1438
 [61]  719 2158 1079 3238 1619 4858 2429 7288 3644 1822  911 2734 1367 4102 2051
 [76] 6154 3077 9232 4616 2308 1154  577 1732  866  433 1300  650  325  976  488
 [91]  244  122   61  184   92   46   23   70   35  106   53  160   80   40   20
[106]   10    5   16    8    4    2    1

Now, let’s suppose that we want to make a dataframe with the sequences for the starting points 1 through 10. The sequence is a vector rather than an integer, so that we need to do this with map:⁷

tibble(start = 1:10) %>% 
  mutate(sequence = map(start, \(start) hotpo_seq(start)))

# A tibble: 10 × 2
   start sequence  
   <int> <list>    
 1     1 <int [1]> 
 2     2 <dbl [2]> 
 3     3 <dbl [8]> 
 4     4 <dbl [3]> 
 5     5 <dbl [6]> 
 6     6 <dbl [9]> 
 7     7 <dbl [17]>
 8     8 <dbl [4]> 
 9     9 <dbl [20]>
10    10 <dbl [7]> 

and we have made a list-column. You can see by the lengths of the vectors in the list-column how long each sequence is.⁸ We might want to make explicit how long each sequence is, and how high it goes:

tibble(start = 1:10) %>% 
  mutate(sequence = map(start, \(start) hotpo_seq(start))) %>% 
  mutate(seq_len = map_int(sequence, \(sequence) length(sequence))) %>% 
  mutate(seq_max = map_int(sequence, \(sequence) max(sequence))) -> seq_info
seq_info

# A tibble: 10 × 4
   start sequence   seq_len seq_max
   <int> <list>       <int>   <int>
 1     1 <int [1]>        1       1
 2     2 <dbl [2]>        2       2
 3     3 <dbl [8]>        8      16
 4     4 <dbl [3]>        3       4
 5     5 <dbl [6]>        6      16
 6     6 <dbl [9]>        9      16
 7     7 <dbl [17]>      17      52
 8     8 <dbl [4]>        4       8
 9     9 <dbl [20]>      20      52
10    10 <dbl [7]>        7      16

To verify for a starting point of 7:

q <- hotpo_seq(7)
q

 [1]  7 22 11 34 17 52 26 13 40 20 10  5 16  8  4  2  1

length(q)

[1] 17

This does indeed have a length of 17 and goes up as high as 52 before coming back down to 1.

Keeping and discarding by name

We don’t have to make a dataframe of these (though that, these days, is usually my preferred way of working). We can instead put the sequences in a list. This one is a “named list”, with each sequence paired with its starting point (its “name”):

seq_list <- seq_info$sequence
names(seq_list) <- seq_info$start
seq_list

$`1`
[1] 1

$`2`
[1] 2 1

$`3`
[1]  3 10  5 16  8  4  2  1

$`4`
[1] 4 2 1

$`5`
[1]  5 16  8  4  2  1

$`6`
[1]  6  3 10  5 16  8  4  2  1

$`7`
 [1]  7 22 11 34 17 52 26 13 40 20 10  5 16  8  4  2  1

$`8`
[1] 8 4 2 1

$`9`
 [1]  9 28 14  7 22 11 34 17 52 26 13 40 20 10  5 16  8  4  2  1

$`10`
[1] 10  5 16  8  4  2  1

If these were in a dataframe as above, a filter would pick out the sequences for particular starting points. As an example, we will pick out the sequences for odd-numbered starting points. Here, this allows us to learn about the new keep_at and discard_at. There is already keep and discard,⁹ for selecting by value, but the new ones allow selecting by name.

There are different ways to use keep_at, but one is to write a function that accepts a name and returns TRUE if that is one of the names you want to keep. Mine is below. The names are text, so I convert the name to an integer and then test it for oddness as we did in hotpo:

keep the sequences for odd-numbered starting points

is_odd <- function(x) {
  x <- as.integer(x)
  x %% 2 == 1
}
is_odd(3)

[1] TRUE

is_odd(4)

[1] FALSE

and now I keep the sequences that have odd starting points thus:

seq_list %>% 
  keep_at(\(x) is_odd(x))

$`1`
[1] 1

$`3`
[1]  3 10  5 16  8  4  2  1

$`5`
[1]  5 16  8  4  2  1

$`7`
 [1]  7 22 11 34 17 52 26 13 40 20 10  5 16  8  4  2  1

$`9`
 [1]  9 28 14  7 22 11 34 17 52 26 13 40 20 10  5 16  8  4  2  1

discard_at selects the ones for which the helper function is FALSE, which in this case will give us the even-numbered starting points:

seq_list %>% 
  discard_at(\(x) is_odd(x))

$`2`
[1] 2 1

$`4`
[1] 4 2 1

$`6`
[1]  6  3 10  5 16  8  4  2  1

$`8`
[1] 8 4 2 1

$`10`
[1] 10  5 16  8  4  2  1

Final thoughts

I have long been a devotee of the lambda-function notation with a map:

x <- 1:5
map_dbl(x, ~sqrt1(.))

[1] 1.000000 1.414214 1.732051 2.000000 2.236068

but I have always had vague misgivings about teaching this, because it is not immediately obvious why the thing inside sqrt1 is not also x. The reason, of course, is the same as this in Python:

x = ['a', 'b', 'c']
for i in x:
  print(i)

a
b
c

where i stands for “the element of x that I am currently looking at”, but it takes a bit of thinking for the learner to get to that point.

Using the anonymous function approach makes things a bit clearer:

x <- 1:5
map_dbl(x, \(x) sqrt1(x))

[1] 1.000000 1.414214 1.732051 2.000000 2.236068

where x appears three times in the map, first as the vector of values of which we want the square roots, and then as the input to sqrt1, so that everything appears to line up.

But there is some sleight of hand here: the meaning of x actually changes as you go along! The first x is a vector, but the second and third x values are numbers, elements of the vector x. Maybe this is all right, because we are used to treating vectors elementwise in R:

tibble(x = 1:5) %>% 
  mutate(root = sqrt(x))

# A tibble: 5 × 2
      x  root
  <int> <dbl>
1     1  1   
2     2  1.41
3     3  1.73
4     4  2   
5     5  2.24

Functions like sqrt are vectorized, so the mutate really means something like “take the elements of x one at a time and take the square root of each one, gluing the result back together into a vector”. So, in the grand scheme of things, I am sold on the (new) anonymous function way of running map, and I think I will be using this rather than the lambda-function way of doing things in the future.

Now, if you’ll excuse me, I now have to attend to all the times I’ve used map in my lecture notes!

Setup

We¹ begin by installing the targets and tarchetypes² packages.

Next, we go down to the console and run

library(targets)

Most of the actual running of things happens in the console with this way of working. Next,

use_targets()

This creates and opens a file called _targets.R that will say how everything is connected together. It is the equivalent of a Makefile in what it does, though the way it works is a bit different from a Makefile. The one use_targets creates is a template, with places to fill in what we need.

I was trying to keep things tidy, so I also created some folders at this point: data, where the data we read in from the website will be stored, report, where the files for my report will live, and R, where the code for our functions will live.

If you are using git/github, there is one more piece of setup to do. targets will create some (potentially) large objects in the folder _targets of your project, and these don’t need to be under version control (because they can always be recreated), so we add the _targets folder to .gitignore.

All of my code is here.

The soap data

The standard way to run targets is to define functions to do everything that needs doing (which you put in one or more files in the R folder), and then edit _targets.R to say how those functions work together to create what you need.

I have three functions: one to read the data from a file, one to make a scatterplot with points and lines defined by groups, and one to fit a regression model, strictly an analysis of covariance model, like this (in file functions.R in folder R):

read_data <- function(filename) {
  read_delim(filename, " ")
}

plot_lines <- function(x) {
  ggplot(x, aes(x = speed, y = scrap, colour = line)) +
    geom_point() +
    geom_smooth(method = "lm")
}

fit_model1 <- function(x) {
  lm(scrap ~ speed + line, data = x)
}

The background is this: our data come from a factory that makes soap bars. The interest here is in how much scrap soap (which cannot be made into soap bars) is produced, which might depend on the speed at which the production is run. There are two different production lines, labelled a and b; the plot suggests that the scrap-speed relationship can be modelled by separate but parallel lines for each production line, which is what the model fits.

Now we have to edit _targets.R to express this. Here is the top bit of mine, with comments edited out:

library(targets)
library(tarchetypes) # Load other packages as needed. # nolint
tar_option_set(
  packages = c("tidyverse"), # packages that your targets need to run
  format = "rds" # default storage format
)
tar_source("R/functions.R")

First we have the targets script load both targets and tarchetypes that we installed earlier.³ Then we add any packages that we need for the analysis to run: in this case tidyverse, or if you’re a stickler for efficiency, readr and ggplot2. The storage format came from use_targets; unless you are creating huge things in your code, the default will be fine. Finally, we load the functions that we wrote.

The bottom bit of _targets.R is the bit that looks like a Makefile, where we say how those functions are going to be used. This is the currently relevant part of mine:

list(
  tar_download(file1, "http://ritsokiguess.site/datafiles/soap.txt",
               paths = "data/soap.txt"),
  tar_target(soap, read_data(file1)),
  tar_target(plot1, plot_lines(soap)),
  tar_target(model1, fit_model1(soap))
)

Each of these create a “target”, the first thing inside tar_target (or tar_download), and anything else says how that target is made. Each tar_target can (and undoubtedly will) use previously made targets.

I have to talk about the first one. My datafile existed on a website with the URL shown, but targets works with local files only. tarchetypes contains a number of recipes for doing jobs like this. tar_download creates here a target file1 that refers to the datafile by downloading the datafile from its website to a file in data. file1 refers to that file without us having to remember where the file is.

So, having created a target file1 that refers to the (downloaded and locally stored) datafile, the next three targets do this:

• read the data in from the local file using our function read_data and store it as the target soap
• make a scatterplot with lines of the soap data, using our function plot_lines and store it as plot1
• fit a model, using our function fit_model, and store it as model1.

Having set up our pipeline, now we can think about running it. Before that, we can go to the console and type tar_visnetwork(). This makes a diagram showing how the bits fit together. In this case we have, left to right:

• file1_url: the url where the data will come from
• file1, which depends on file1_url
• soap, which depends on file1
• plot1, which depends on soap
• model1, which also depends on soap.

The value of looking at this diagram is to make sure that we have coded the dependency structure properly, which it seems (in this case) I have done.

Also note that the targets are colour-coded according to whether they are up-to-date or outdated. At first, everything will be outdated, but later only some of it will be outdated. targets knows to only update what needs updating.

Next, we actually run everything. This is done in the console with tar_make. This will update everything that needs updating, using in each case the recipe in _targets.R. The output to tar_make() tells you whether each target was “built” (updated) or “skipped” (nothing in that target had changed). If anything goes wrong, there will be an error message. I find the error messages not very helpful, but at least it is clear which of the targets caused the error, which at least gives a place to start looking.

Once everything has run, the output from each target is stored,⁴ and can be inspected (in the Console) using tar_read. For example, tar_read(soap) will display the dataframe that was read in from the file, and tar_read(plot1) will display the scatterplot. You can do the same thing with the fitted model object model1, but this is probably not what you want; you would probably prefer to look at the summary of the model,⁵ which you can do like this:

tar_load(model1)
summary(model1)

tar_load puts a copy of the target named into your workspace, and then you can do something with it as you normally would.

Making a child report

So far, this is very standard targets work: use functions to make a pipeline that you specify in _targets.R, and then use tar_make to run it. But, having gotten this to work, I wanted to add a report, and I wanted to do so flexibly, so that I could easily add a different analysis of a different dataset later. The way I like to do this is to use “child documents”: write each report as a separate .Rmd file, and then have a “parent” report that loads each child report.

We’ll get to the parent report later. Let’s write a report about the soap data first, which will be saved in soap.Rmd in the report folder. This is a child document, so it has no YAML header, and we begin right away with the report header and a description of the dataset. The report structure will be simple: we display the data, display the scatterplot (and talk about it a bit), then display the regression output (and talk about that a bit).

There is a standard targets way of making reports like these: we do all the computation in previous targets (as we have done), and then we read in what we want to display with tar_read or tar_load as we did above, instead of doing more computation to obtain a target that we already have. For this report, that means having three code chunks, the content of which we have already seen:

tar_read(soap)

to display the data,

tar_read(plot1)

to display the scatterplot, and

tar_load(model1)
summary(model1)

to display the model output. That, together with my comments, is soap.Rmd.

Finally, we need to add this into the pipeline, so that targets knows to update this part of the report if the text in it changes, or if any of soap, plot1, or model1 changes.⁶ This is a so-called “file” target, and we add it to the end of _targets.R to make this:

list(
  tar_download(file1, "http://ritsokiguess.site/datafiles/soap.txt",
               paths = "data/soap.txt"),
  tar_target(soap, read_data(file1)),
  tar_target(plot1, plot_lines(soap)),
  tar_target(model1, fit_model1(soap)),
  tar_target(soap_report, "report/soap.Rmd", format = "file")
)

targets knows additionally that the soap report depends on soap, plot1, and model1 because of the tar_read and tar_load statements in the report. (This doesn’t always show up in tar_visnetwork, but targets knows about it all the same.)

Making a parent report

This is a genuine Markdown report, so it begins with a YAML header specifying the author, title, date, etc. Then follows a setup chunk with knitr::opts_chunk$set(echo = FALSE): the only code in this part of the report is the tar_read and tar_load statements that the reader doesn’t need to see.

Then we need to say that this report depends on soap_report, so that if that changes, the parent report needs to be updated. I come from a Makefile background, so this took some time (and help) for me to figure out, but the way you do it is, as in the child report, tar_loading the target that represents the report:

tar_load(soap_report)

Then I had some preamble text.

Then we load the child report itself. It seems that it should be possible to use soap_report directly (it contains the path to the child report), but I couldn’t get this to work, so I specified the actual path to the child report directly with child = "report/soap.Rmd" as a chunk option.⁷

The last thing to do here is to add the parent report as a target. We now have:

list(
  tar_download(file1, "http://ritsokiguess.site/datafiles/soap.txt",
               paths = "data/soap.txt"),
  tar_target(soap, read_data(file1)),
  tar_target(plot1, plot_lines(soap)),
  tar_target(model1, fit_model1(soap)),
  tar_target(soap_report, "report/soap.Rmd", format = "file"),
  tar_render(final_report, "report/report.Rmd")
)

The parent report is the thing that needs to be knitted, so we use the special target tar_render (from tarchetypes), which says to take the document stored in the second input, and knit it to create the target that is the first input. After this runs, there is a file report.html in report that is the knitted report.

Including the code in a report

If I were writing the report for someone else, I wouldn’t expect them to be very interested in the code that produced the plot or the model summary. But for teaching, it is very useful to show what code the output came from. The problem with the targets-style analysis we just did is that, by separating the computation from the reporting, the code is nowhere to be found.

At the expense of good targets style and efficient computation, however, there is no problem including the computation in the report, so that my second child report, in report/spiders.Rmd, looks like any other R Notebook you might write, with code chunks and the output immediately below them, to read in the data from a website, make a boxplot and run a logistic regression. The background for this one is that a certain spider is or is not found on a beach, and the research hypothesis is that whether or not this spider is found depends somehow on the size of the grains of sand on that beach.

So there is no difficulty composing this kind of report, and no need to write extra functions and make extra targets to compute its constituent pieces. The only extra thing that needs to go in _targets.R is this:

list(
  tar_download(file1, "http://ritsokiguess.site/datafiles/soap.txt",
               paths = "data/soap.txt"),
  tar_target(soap, read_data(file1)),
  tar_target(plot1, plot_lines(soap)),
  tar_target(model1, fit_model1(soap)),
  tar_target(soap_report, "report/soap.Rmd", format = "file"),
  tar_target(spiders_report, "report/spiders.Rmd", format = "file"),
  tar_render(final_report, "report/report.Rmd")
)

(the second to last line, entirely analogous to the other child report). The extra stuff that goes in the parent report is to turn the display of code back on:

knitr::opts_chunk$set(echo = TRUE)

Another tar_load:

tar_load(spiders_report)

(to build the dependence of the parent report on this child one as well), and then the actual importation of the second child report with child = "report/spiders.Rmd" in the options of another empty code chunk.

This puts all the dependencies in the right places, and so another tar_make will build the whole report with its two child reports on the two datasets, updating anything that needs updating.

Introduction

In a previous post, I discussed how we might sample in groups, where each group was a sample from a different population.

I introduced this function:

gen_sample <- function(n, mean, sd) {
  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z) %>% 
    select(gp, z)
}

that samples from normal populations with possibly different means, SDs, and sample sizes in different groups.

Explanation

This is (more or less) the same explanation that appeared at the end of the previous post, so feel free to skip if you have read it before.

The first step is to make a data frame with one row for each sample that will be generated. This uses the inputs to the function above, so we will make some up:

n <- c(5, 3)
mean <- c(20, 10)
sd <- c(2, 1)
tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) 

# A tibble: 2 × 4
  gp        n  mean    sd
  <chr> <dbl> <dbl> <dbl>
1 x         5    20     2
2 y         3    10     1

Evidently, in a function for public consumption, you would check that all the inputs are the same length, or you would rely on tibble telling you that only vectors of length 1 are recycled.¹ The groups are for no good reason called x and y.

The next two lines generate random samples, one for each group, according to the specifications, and store them each in one cell of the two-row spreadsheet:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) 

# A tibble: 2 × 5
# Rowwise: 
  gp        n  mean    sd z        
  <chr> <dbl> <dbl> <dbl> <list>   
1 x         5    20     2 <dbl [5]>
2 y         3    10     1 <dbl [3]>

The new column z is a list column, since the top cell of the column is a vector of length 5, and the bottom cell is a vector of length 3. To actually see the values they contain, we unnest z:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z)

# A tibble: 8 × 5
  gp        n  mean    sd     z
  <chr> <dbl> <dbl> <dbl> <dbl>
1 x         5    20     2 21.9 
2 x         5    20     2 20.1 
3 x         5    20     2 18.8 
4 x         5    20     2 20.4 
5 x         5    20     2 15.8 
6 y         3    10     1  9.74
7 y         3    10     1 11.0 
8 y         3    10     1 10.2 

and, finally, the middle three columns were only used to generate the values in z, so they can be thrown away now by selecting only gp and z.

The rowwise is necessary:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z)

# A tibble: 4 × 5
  gp        n  mean    sd     z
  <chr> <dbl> <dbl> <dbl> <dbl>
1 x         5    20     2 22.7 
2 x         5    20     2  9.92
3 y         3    10     1 22.7 
4 y         3    10     1  9.92

because rnorm is vectorized, and for the x sample, R will draw one sampled value from each normal distribution, and then repeat the same values for the y sample. This is very much not what we want.

The same idea can be used to draw random chi-squared data (say):

tibble(df = c(2,6)) %>% 
  rowwise() %>% 
  mutate(z = list(rchisq(5, df))) %>% 
  unnest(z)

# A tibble: 10 × 2
      df       z
   <dbl>   <dbl>
 1     2  0.298 
 2     2  1.11  
 3     2  0.445 
 4     2  1.11  
 5     2  0.0144
 6     6  7.54  
 7     6  4.46  
 8     6 14.8   
 9     6  9.13  
10     6  7.99  

(five values from , followed by five from .)

This suggests that I ought to be able to generalize my function gen_sample. Generalizing to any number of groups needs no extra work: the length of the input n determines the number of groups, and the values in n determine the size of each of those groups.

The interesting generalization is the distribution to sample from. The first parameter of the functions rnorm, rchisq etc. is always the number of random values to generate, but the remaining parameters are different for each distribution. This suggests that my generalized random sample generator ought to have the name of the random sampling function as input, followed by some means to allow any other inputs needed by that sampling function.

Generalizing

To generalize gen_sample to a new function sample_groups, we need to consider how to handle different distributions. The distribution itself is not the hard part; that can be specified by having the random sample generator function for the desired distribution as an input to the new function. The problem is that each distribution has different parameters, which need to be inputs to sample_groups.

The standard way of doing this kind of thing is to use the ... input to a function. If I had just one group, it would go like this:

sample_group <- function(n, dist, ...) {
  dist(n, ...)
}

and then

sample_group(5, rnorm, mean = 10, sd = 3)

[1]  8.634868  8.737404  7.642048 10.836333  9.608791

or

sample_group(6, rpois, lambda = 2)

[1] 3 6 1 1 1 0

The additional (named) inputs to sample_group are passed on unchanged to the random sample generator, to generate respectively normal random values with mean 10 and SD 3, and Poisson random values with mean 2.²

The random sample generators are vectorized, but the obvious thing for generating two samples from different distributions does not work:

sample_group(c(6, 3), rnorm, mean = c(10, 20), sd = c(3, 2))

[1]  7.124158 21.351194

We appear to have one value from each distribution, not six from the first and three from the second. This, I think, is what the help files say will happen.

To allow for the stuff at the end of the call to be different, another way is to use a list to pass each distribution’s parameters. This turns out to be what I do later.

A bad approach

Let’s suppose I ask the user to write the code to generate each sample as text (a vector of pieces of text, one for each sample). Here’s how my example above would look:

code <- c("rnorm(5, mean = 10, sd = 3)", 
          "rpois(6, lambda = 2)")
code

[1] "rnorm(5, mean = 10, sd = 3)" "rpois(6, lambda = 2)"       

The problem is that this is text, not runnable code. One way to turn this into something useful is to parse it:

pc <- parse(text = code[1])
pc

expression(rnorm(5, mean = 10, sd = 3))

This has turned the text into an expression, something that can be evaluated, thus:

eval(pc)

[1]  9.273030  8.994949 15.743894 11.803060 13.231719

And now we have a strategy:

tibble(code) %>% 
  rowwise() %>% 
  mutate(expr = list(parse(text = code))) %>%
  mutate(z = list(eval(expr))) %>% 
  unnest(z)

# A tibble: 11 × 3
   code                        expr               z
   <chr>                       <list>         <dbl>
 1 rnorm(5, mean = 10, sd = 3) <expression>  9.57  
 2 rnorm(5, mean = 10, sd = 3) <expression> -0.0996
 3 rnorm(5, mean = 10, sd = 3) <expression>  6.94  
 4 rnorm(5, mean = 10, sd = 3) <expression> 15.5   
 5 rnorm(5, mean = 10, sd = 3) <expression>  9.09  
 6 rpois(6, lambda = 2)        <expression>  2     
 7 rpois(6, lambda = 2)        <expression>  4     
 8 rpois(6, lambda = 2)        <expression>  6     
 9 rpois(6, lambda = 2)        <expression>  0     
10 rpois(6, lambda = 2)        <expression>  4     
11 rpois(6, lambda = 2)        <expression>  3     

So now we have the ingredients for a version of sample_groups based on the user writing the random-sampling code for us. I added one extra thing: lettering the groups, since they otherwise have bad names:

sample_groups <- function(code) {
  n_pop <- length(code)
  tibble(code, gp = letters[1:n_pop]) %>% 
    rowwise() %>% 
    mutate(expr = list(parse(text = code))) %>%
    mutate(z = list(eval(expr))) %>% 
    unnest(z) %>% 
    select(gp, z)
}

and to test:

d <- sample_groups(code)
d

# A tibble: 11 × 2
   gp        z
   <chr> <dbl>
 1 a      7.81
 2 a     10.3 
 3 a      7.07
 4 a     10.1 
 5 a      8.93
 6 b      2   
 7 b      1   
 8 b      2   
 9 b      1   
10 b      6   
11 b      2   

Using the eval(parse(text = something)) idea is not (apparently) very well regarded.³ One immediate problem is that the user could put any code at all (that evaluates into a vector of numbers) into the input code, which seems less than secure:

code <- c("1:3", "mtcars$mpg")
sample_groups(code)

# A tibble: 35 × 2
   gp        z
   <chr> <dbl>
 1 a       1  
 2 a       2  
 3 a       3  
 4 b      21  
 5 b      21  
 6 b      22.8
 7 b      21.4
 8 b      18.7
 9 b      18.1
10 b      14.3
# ℹ 25 more rows

A better way

I want to get back to the user inputting the desired random sample generators as functions, and then running those functions on the rest of the input. This is what do.call does:

do.call(rnorm, list(n = 5, mean = 10, sd = 3))

[1]  7.493600 13.180428  8.346022  8.472318 11.365156

do.call(rpois, list(n = 6, lambda = 2))

[1] 2 1 4 1 2 1

Having realized (i) that do.call is what I wanted, and (ii) that the input parameters to the functions need to be in a list, I packaged up those distribution parameters into a list of lists first. It is actually not necessary to make the list of distributions into a list, but it works if you do that too:

dist <- list(rnorm, rpois)
pars <- list(list(n = 5, mean = 10, sd = 3), 
             list(n = 6, lambda = 2))
d <- tibble(dist = dist, pars = pars)
d

# A tibble: 2 × 2
  dist   pars            
  <list> <list>          
1 <fn>   <named list [3]>
2 <fn>   <named list [2]>

and then we put the do.call in a rowwise mutate, wrapping the whole thing in a list to make a list-column:

d %>% 
  rowwise() %>% 
  mutate(z = list(do.call(dist, pars))) %>% 
  unnest(z)

# A tibble: 11 × 3
   dist   pars                 z
   <list> <list>           <dbl>
 1 <fn>   <named list [3]>  5.97
 2 <fn>   <named list [3]> 12.0 
 3 <fn>   <named list [3]> 10.1 
 4 <fn>   <named list [3]> 18.3 
 5 <fn>   <named list [3]>  6.27
 6 <fn>   <named list [2]>  3   
 7 <fn>   <named list [2]>  2   
 8 <fn>   <named list [2]>  2   
 9 <fn>   <named list [2]>  2   
10 <fn>   <named list [2]>  2   
11 <fn>   <named list [2]>  2   

it works!

And so, we can now make our function:

sample_groups <- function(dist, pars) {
  nr <- length(pars)
  tibble(dist = dist, pars = pars, gp = letters[1:nr]) %>% 
    rowwise() %>% 
    mutate(z = list(do.call(dist, pars))) %>% 
    unnest(z) %>% 
    select(gp, z)
}

and to test

dists <- list(rnorm, rpois)
pars <- list(list(n = 5, mean = 10, sd = 3), list(n = 6, lambda = 2))
sample_groups(dists, pars)

# A tibble: 11 × 2
   gp        z
   <chr> <dbl>
 1 a     13.6 
 2 a     10.7 
 3 a     14.3 
 4 a      7.93
 5 a      6.76
 6 b      2   
 7 b      0   
 8 b      2   
 9 b      4   
10 b      3   
11 b      3   

The only weirdness is that the user has to specify a list of lists for the parameters (because do.call needs a list for inputs to its function). But it definitely works.

One shortcut is that if you want all the samples to be from the same distribution, you specify only one thing in the input that I called dists:⁴

dist <- list(rnorm)
new_pars <- list(list(n = 3, mean = 5, sd = 1),
                 list(n = 2, mean = 10, sd = 2),
                 list(n = 4, mean = 20, sd = 3))
sample_groups(dist, pars = new_pars)

# A tibble: 9 × 2
  gp        z
  <chr> <dbl>
1 a      3.58
2 a      5.26
3 a      4.75
4 b      8.15
5 b     12.5 
6 c     21.1 
7 c     17.7 
8 c     17.7 
9 c     14.6 

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(smmr)

The package smmr is one that I wrote myself, and lives on Github here. It may be installed using install_github from package remotes, or using pkg_install from package pak.

The two-sample -test

The two-sample -test is probably your first idea for comparing (means of) two independent samples from two different populations, such as one set of people who undergo some (real) treatment and a different set of people who undergo some different (eg. placebo) treatment.¹

There are actually two flavours of two-sample -test:

• If you took a mathematical statistics class, you most likely learned about the pooled test. Here, as well as the normality assumption that we talk about later, it is assumed that the two populations have the same variance, which makes the algebra a lot easier. (To run the test, we estimate the common variance by making a weighted average of the two sample variances.)
• The default used by R’s t.test is a different test due to Welch and to Satterthwaite.² This does not assume equal variances, but the test statistic (obtained by estimating each population’s variance separately) does not have an exact -distribution. It is assumed that the test statistic has a -distribution with a different (usually fractional) degrees of freedom that Welch and Satterthwaite each give a formula for; this assumption is usually good. The Wikipedia page gives the formula and references to the original papers (in Further Reading).

In practice, when both tests apply, the two tests usually give very similar P-values, and so there is no harm in only ever using the Welch-Satterthwaite test, and thus the pooled -test is really only a curiosity of math stat classes. If you really want the pooled test, you have to ask t.test for it specifically (using var.equal = TRUE). If, however, the two populations have different variances, then the pooled test can be very misleading.

I will illustrate, first from populations with equal variances. Before that, though, generating the random samples in long format is a bit annoying, so I define a function to do that first, with the sample sizes, population means, and population SDs as vector input (each of length 2):³

gen_sample <- function(n, mean, sd) {
  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z) %>% 
    select(gp, z)
}

The simulated data values are called z and the groups are called x and y in column gp. As written, the data are always drawn from a normal distribution.

Now we generate some data and run the two flavours of -test on it. This time, the means are different (so the null is actually false) but the SDs are the same:

d1 <- gen_sample(n = c(10, 10), mean = c(100, 140), sd = c(30, 30))
t.test(z ~ gp, data = d1)

    Welch Two Sample t-test

data:  z by gp
t = -2.2065, df = 16.785, p-value = 0.04158
alternative hypothesis: true difference in means between group x and group y is not equal to 0
95 percent confidence interval:
 -59.836964  -1.311008
sample estimates:
mean in group x mean in group y 
       104.9265        135.5005 

t.test(z ~ gp, data = d1, var.equal = TRUE)

    Two Sample t-test

data:  z by gp
t = -2.2065, df = 18, p-value = 0.04058
alternative hypothesis: true difference in means between group x and group y is not equal to 0
95 percent confidence interval:
 -59.685224  -1.462748
sample estimates:
mean in group x mean in group y 
       104.9265        135.5005 

In this case, the P-values are almost identical.

Let’s try it again, but this time group has a smaller sample and also a larger variance:

d2 <- gen_sample(c(10, 5), c(100, 140), c(30, 60))
t.test(z ~ gp, data = d2)

    Welch Two Sample t-test

data:  z by gp
t = -1.6394, df = 4.4934, p-value = 0.1686
alternative hypothesis: true difference in means between group x and group y is not equal to 0
95 percent confidence interval:
 -127.22157   30.20689
sample estimates:
mean in group x mean in group y 
       106.2476        154.7549 

t.test(z ~ gp, data = d2, var.equal = TRUE)

    Two Sample t-test

data:  z by gp
t = -2.2024, df = 13, p-value = 0.04629
alternative hypothesis: true difference in means between group x and group y is not equal to 0
95 percent confidence interval:
 -96.0887846  -0.9258949
sample estimates:
mean in group x mean in group y 
       106.2476        154.7549 

This time, the pooled test has a much smaller P-value, which makes us think there is a real difference between the means, even though the (correct⁴) Welch test says that the evidence is not strong enough, probably because the sample sizes are not big enough.

If the smaller sample had come from the population with smaller variance, things would not have been so bad estimation-wise, but having the small sample be less informative about its population mean is asking for trouble.

The same derivation that Welch used applies to a comparison of any number of groups, so there is also a Welch ANOVA for comparing the means of three or more groups, without assuming equal variances. Likewise, the -statistic no longer has exactly an distribution, so Welch obtained an approximate denominator degrees of freedom so that the -test is still good enough. R has this in oneway.test. Welch’s ANOVA deserves to be a lot better known than it is.⁵

The two-sample -tests have a normality assumption, like the one-sample . Here, it is that the observations from each population have a normal distribution, independently of each other and the observations from the other population. As with a one-sample test, the Central Limit Theorem helps, and with larger sample sizes, the normality matters less. I tend to say that each sample should be close enough to normal in shape given its sample size (as you might assess with separate normal quantile plots for each sample), but this is being somewhat too stringent because the statistic for either of the two-sample tests is based on the difference between the sample means, and that will tend to be a bit more normal than either sampling distribution of the two sample means individually. You might assess this with a bootstrap distribution of the -statistic (or of the difference in sample means), though this requires care to get bootstrap samples of the same size as the original ones (simply resampling rows of a long data frame will not do this).

The rank sum test

So, what to do if the observations within each sample are not as normal as you would like? Something that is often suggested is the rank sum test, often with the names Mann and Whitney attached, and sometimes with the same name Wilcoxon that is attached to the signed rank test. I illustrate with the same data I used for the second version of the two-sample :

wilcox.test(z ~ gp, data = d2)

    Wilcoxon rank sum exact test

data:  z by gp
W = 15, p-value = 0.2544
alternative hypothesis: true location shift is not equal to 0

To see where the W = 15 came from:

d2 %>% mutate(rk = rank(z)) %>% 
  group_by(gp) %>% 
  summarize(rank_sum = sum(rk),
            n = n()) -> s1
s1

# A tibble: 2 × 3
  gp    rank_sum     n
  <chr>    <dbl> <int>
1 x           70    10
2 y           50     5

The two groups are pooled together and ranked from smallest to largest, and then the ranks for each group are summed. There are only 5 observations in group y, so the ranks in this group are typically larger (to go with the values themselves being typically larger). To account for this, the following calculation is done:

s1 %>% 
  rowwise() %>% 
  mutate(W = rank_sum - n*(n+1)/2)

# A tibble: 2 × 4
# Rowwise: 
  gp    rank_sum     n     W
  <chr>    <dbl> <int> <dbl>
1 x           70    10    15
2 y           50     5    35

and the smaller of the two values in W is the test statistic. The smaller the test statistic is, the more significant (the smallest possible value is zero). In this case, the P-value is 0.2544, not significant.

What happens if one of the samples is more variable, even if the means and sample sizes are the same, so that the null hypothesis is still true? We should therefore reject 5% of the time still. Let’s do a simulation to find out:⁶

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(my_sample = list(gen_sample(n = c(10, 10), mean = c(100, 100), sd = c(30, 5)))) %>% 
  mutate(my_test = list(wilcox.test(z ~ gp, data = my_sample))) %>% 
  mutate(p_val = my_test$p.value) -> d5
d5

# A tibble: 10,000 × 4
# Rowwise: 
     sim my_sample         my_test p_val
   <int> <list>            <list>  <dbl>
 1     1 <tibble [20 × 2]> <htest> 0.971
 2     2 <tibble [20 × 2]> <htest> 0.280
 3     3 <tibble [20 × 2]> <htest> 0.684
 4     4 <tibble [20 × 2]> <htest> 0.579
 5     5 <tibble [20 × 2]> <htest> 0.853
 6     6 <tibble [20 × 2]> <htest> 0.481
 7     7 <tibble [20 × 2]> <htest> 0.684
 8     8 <tibble [20 × 2]> <htest> 0.190
 9     9 <tibble [20 × 2]> <htest> 0.971
10    10 <tibble [20 × 2]> <htest> 0.247
# ℹ 9,990 more rows

How many of these P-values are 0.05 or less?

d5 %>% count(p_val <= 0.05)

# A tibble: 2 × 2
# Rowwise: 
  `p_val <= 0.05`     n
  <lgl>           <int>
1 FALSE            9193
2 TRUE              807

This says: we are rejecting over 8% of the time, with a test that is supposed to reject only 5% of the time. The reason for doing 10,000 simulations is so that we can get a good sense of whether this is “really” greater than 5%:

prop.test(824, 10000, p = 0.05)

    1-sample proportions test with continuity correction

data:  824 out of 10000, null probability 0.05
X-squared = 220.32, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.05
95 percent confidence interval:
 0.07712114 0.08800255
sample estimates:
     p 
0.0824 

The probability of incorrectly rejecting the true null is definitely not 0.05, and the confidence interval indicates that it is substantially greater than 0.05. So we should not be using the rank sum test if the two populations have different variances: in other words, the rank sum test suffers from the same problems as the pooled -test.

This is often stated as saying that the rank sum test is actually testing a null hypothesis of equal distributions, and if you reject, as you too often do here, the distributions could differ in some way other than equal means. This might be what you want (though, as this simulation shows, you do not get much power to detect unequal spreads), but in the kind of situation where you would have done a -test, it most likely is not. We don’t want to be worrying about whether spreads or distribution shapes differ when our principal interest is in means or medians.

Mood’s median test

So, if the rank sum test doesn’t do the job when the -test doesn’t, what does? I suggest a test that seems to be unjustly maligned called Mood’s median test. It is a sort of two-sample version of the sign test, and like the sign test, it is a test for medians.⁷

To illustrate, let’s generate some data from right-skewed chi-squared distributions: one sample with 2 df (that has mean 2) and one sample with 6 df (that has mean 6):

tibble(df = c(2,6)) %>% 
  rowwise() %>% 
  mutate(z = list(rchisq(20, df))) %>% 
  unnest(z) -> d6
ggplot(d6, aes(x = z)) + geom_histogram(bins = 5) +
  facet_wrap(~df, ncol = 1)

At least the first of these does not look very normal (dissuading us from a -test), and they don’t seem to have the same spread or shape of distribution (dissuading us from a rank sum test).

The idea behind the test is to work out the median of all the data, and then to count the number of observations above and below this grand median. This is much as you would do for a sign test, but here we count aboves and belows for ecah group separately:⁸

med <- median(d6$z)
med 

[1] 3.522339

tab <- with(d6, table(group = df, above = (z > med)))
tab

     above
group FALSE TRUE
    2    15    5
    6     5   15

If the two groups have the same median, about 50% of the observations in each group should be above the overall median and about 50% below. If the two groups have different medians, one of the groups will have most of its observations above the grand median, and the other one will have most of its observations below. As for the sign test, it doesn’t matter how far above or below the grand median each observation is, just whether it is above or below.

In the example above, knowing which group an observation is from tells you something about whether it is likely to be above or below the grand median (if the 2 df group, probably below; if the 6 df group, probably above). Hence there appears to be an association between group and being above or below, and you might imagine testing this with a chi-squared test for association. This is how I run the test in my smmr package:

median_test(d6, z, df)

$grand_median
[1] 3.522339

$table
     above
group above below
    2     5    15
    6    15     5

$test
       what        value
1 statistic 10.000000000
2        df  1.000000000
3   P-value  0.001565402

The P-value is definitely small enough to conclude that there is an association, and hence to (correctly) conclude that the two groups have different medians.

A couple of technicalities:

• This runs the chi-squared test without Yates’ correction, even for 2-by-2 tables. This is to enable you to get the same result by hand-calculation, should you remember how to do that. The chisq.test function does by default use this correction for 2-by-2 tables (see the help for chisq.test).
• You might have observed in this example that the row totals are fixed (each sample is of size 20, split between above and below the grand median somehow), and also the column totals are fixed (altogether 20 of the data values must be above the grand median and 20 below). This is always true, and so you might consider running Fisher’s exact test here instead, which I do below. I didn’t put this in my package, for a number of reasons: (i) I would have to teach my students Fisher’s exact test first; (ii) depending on the level of the class, I would also have to teach the hypergeometric distribution; (iii) Mood’s median test also applies to more than two groups (see below), but it is not clear to me that Fisher’s exact test applies to the by 2 table you would get from groups.

Fisher’s exact test for the same data looks like this:

fisher.test(tab)

    Fisher's Exact Test for Count Data

data:  tab
p-value = 0.003848
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
  1.794006 48.338244
sample estimates:
odds ratio 
  8.418596 

The P-value is bigger here, but still significant. End of technicalities.

As the sign test does, this test counts only whether each data value is above or below something, and this is using the data inefficiently if the actual values are meaningful. Thus you would expect the -test to be more powerful if it is valid, but this is of no concern, because in that case you would use the -test. When the -test is not valid, Mood’s median test makes no assumptions about the data (in contrast to the rank sum test): if the two populations have the same median, about 50% of the values in each group will be above that common median, and if they don’t, there will be an association between group and being above/below that the chi-squared test has a chance at finding. This is regardless of the shape of either distribution.

More than two groups: ANOVA revisited

You may have noticed that it doesn’t really matter how many groups you have: you work out the median of all the observations and count above and below within each group, no matter how many groups there are. The null hypothesis in this case is that all the groups have the same median, and the alternative is “not the null”. This is analogous to one-way ANOVA, where the null hypothesis is that all the groups have the same mean, and if rejected, there is further work to do to find which groups differ from which. You might do that with Tukey’s method.

You might use Mood’s median test in an ANOVA-type situation where you felt that the observations within each group were not normal enough given your sample sizes. Since this test is analogous to the -test, you may need a followup to decide which groups have different medians. One way to do this is to run Mood’s median test on all possible pairs of groups (ignoring the data in groups other than the ones you are comparing), and then do an adjustment to the P-values like Bonferroni or Holm to account for the multiple testing.

I would actually go further than this. I would begin by drawing a boxplot to assess normality and equal spreads within each group, and then:

• if the data are normal enough and the spreads⁹ are more or less equal, do ordinary ANOVA followed by Tukey if needed.
• if the data are normal enough but the spreads are not more or less equal, do Welch’s ANOVA (see above) using oneway.test, following up if needed with Games-Howell. The Games-Howell procedure is available in package PMCMRplus as gamesHowellTest.
• if the data are not normal enough, do Mood’s median test, followed by pairwise Mood’s median tests, adjusting for multiple testing.

All of this seems to need an example. I use the InsectSprays data. These are counts of insects in experimental units treated with different insecticides. The fact that these are counts suggests that higher counts might be more variable:

data("InsectSprays")

As suggested above, we start with boxplots:

ggplot(InsectSprays, aes(x = spray, y = count)) + geom_boxplot()

The smaller counts (associated with sprays C, D, and E) do seem to be less variable. The normality is mostly not too bad, though there are some high outliers with sprays C and E. (There are only twelve observations for each spray.)

The counts do not have a common spread across sprays, so ordinary ANOVA is out of the question, but Welch ANOVA might be OK:

oneway.test(count ~ spray, data = InsectSprays)

    One-way analysis of means (not assuming equal variances)

data:  count and spray
F = 36.065, num df = 5.000, denom df = 30.043, p-value = 7.999e-12

Strongly significant, so we need some kind of post-hoc test. The recommended one is called Games-Howell,¹⁰ which can be found in the PMCMRplus package:

PMCMRplus::gamesHowellTest(count ~ spray, data = InsectSprays)

    Pairwise comparisons using Games-Howell test

data: count by spray

  A       B       C       D       E      
B 0.99725 -       -       -       -      
C 6.6e-06 6.7e-07 -       -       -      
D 0.00013 1.3e-05 0.05567 -       -      
E 3.2e-05 3.7e-06 0.44661 0.60060 -      
F 0.92475 0.98879 3.4e-05 0.00029 0.00011

P value adjustment method: none

alternative hypothesis: two.sided

A look at the boxplot suggests that the sprays divide into two sets: A, B, F (high insect count), and C, D, E (low). This is how Games-Howell comes out, though the C-D difference is almost significant.

If you are bothered by the outliers, then Mood’s median test is the way to go (from smmr):

median_test(InsectSprays, count, spray)

$grand_median
[1] 7

$table
     above
group above below
    A    11     0
    B    11     0
    C     0    11
    D     1    11
    E     0    12
    F    12     0

$test
       what        value
1 statistic 6.533256e+01
2        df 5.000000e+00
3   P-value 9.561214e-13

This is also strongly significant. Looking at the table of values above and below suggests the same division of the sprays into two sets:

pairwise_median_test(InsectSprays, count, spray)

# A tibble: 15 × 4
   g1    g2        p_value adj_p_value
   <fct> <fct>       <dbl>       <dbl>
 1 A     B     0.391         1        
 2 A     C     0.00000273    0.0000409
 3 A     D     0.0000446     0.000668 
 4 A     E     0.000000963   0.0000145
 5 A     F     0.528         1        
 6 B     C     0.00000273    0.0000409
 7 B     D     0.0000446     0.000668 
 8 B     E     0.000000963   0.0000145
 9 B     F     0.414         1        
10 C     D     0.00165       0.0248   
11 C     E     0.0661        0.991    
12 C     F     0.000000963   0.0000145
13 D     E     0.123         1        
14 D     F     0.0000446     0.000668 
15 E     F     0.000000963   0.0000145

though this time sprays C and D are significantly different, with a P-value of 0.025.

Appendix: generating samples from groups

Earlier, I threw this function at you without explaining it:

gen_sample

function (n, mean, sd) 
{
    tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
        rowwise() %>% mutate(z = list(rnorm(n, mean, sd))) %>% 
        unnest(z) %>% select(gp, z)
}
<bytecode: 0x5b4269900df8>

There are different ways to generate samples from different groups with possibly different means, SDs and sample sizes. This is how I like to do it. Let me take you through the process.

The first step is to make a data frame with one row for each sample that will be generated. This uses the inputs to the function above, so we will make some up:

n <- c(5, 3)
mean <- c(20, 10)
sd <- c(2, 1)
tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) 

# A tibble: 2 × 4
  gp        n  mean    sd
  <chr> <dbl> <dbl> <dbl>
1 x         5    20     2
2 y         3    10     1

Evidently, in a function for public consumption, you would check that all the inputs are the same length, or you would rely on tibble telling you that only vectors of length 1 are recycled.¹¹ The groups are for no good reason called x and y.

The next two lines generate random samples, one for each group, according to the specifications, and store them each in one cell of the two-row spreadsheet:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) 

# A tibble: 2 × 5
# Rowwise: 
  gp        n  mean    sd z        
  <chr> <dbl> <dbl> <dbl> <list>   
1 x         5    20     2 <dbl [5]>
2 y         3    10     1 <dbl [3]>

The new column z is a list column, since the top cell of the column is a vector of length 5, and the bottom cell is a vector of length 3. To actually see the values they contain, we unnest z:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    rowwise() %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z)

# A tibble: 8 × 5
  gp        n  mean    sd     z
  <chr> <dbl> <dbl> <dbl> <dbl>
1 x         5    20     2 23.1 
2 x         5    20     2 22.2 
3 x         5    20     2 20.6 
4 x         5    20     2 17.0 
5 x         5    20     2 21.1 
6 y         3    10     1 10.7 
7 y         3    10     1 10.1 
8 y         3    10     1  9.25

and, finally, the middle three columns were only used to generate the values in z, so they can be thrown away now by selecting only gp and z.

The rowwise is necessary:

  tibble(gp = c("x", "y"), n = n, mean = mean, sd = sd) %>% 
    mutate(z = list(rnorm(n, mean, sd))) %>% 
    unnest(z)

# A tibble: 4 × 5
  gp        n  mean    sd     z
  <chr> <dbl> <dbl> <dbl> <dbl>
1 x         5    20     2 15.3 
2 x         5    20     2  9.45
3 y         3    10     1 15.3 
4 y         3    10     1  9.45

because rnorm is vectorized, and for the x sample, R will draw one sampled value from each normal distribution, and then repeat the same values for the y sample. This is very much not what we want.

I used the same idea to draw my random chi-squared data later on:

tibble(df = c(2,6)) %>% 
  rowwise() %>% 
  mutate(z = list(rchisq(20, df))) %>% 
  unnest(z)

# A tibble: 40 × 2
      df      z
   <dbl>  <dbl>
 1     2  2.23 
 2     2  3.58 
 3     2  1.56 
 4     2 11.1  
 5     2  0.924
 6     2  0.263
 7     2  0.692
 8     2  2.81 
 9     2  4.85 
10     2  2.39 
# ℹ 30 more rows

(twenty values from , followed by twenty from .)

This suggests that I ought to be able to generalize my function gen_sample. Generalizing to any number of groups needs no extra work: the length of the input n determines the number of groups, and the values in n determine the size of each of those groups.

The interesting generalization is the distribution to sample from. The first parameter of the functions rnorm, rchisq etc. is always the number of random values to generate, but the remaining parameters are different for each distribution. This suggests that my generalized random sample generator ought to have the name of the random sampling function as input, followed by ... to allow any other inputs needed by that sampling function; these then get passed on. At present, this idea is still living in my head, so I think I need to write another blog post about that to make sure that it does indeed work.

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

"Screenshot 2025-12-28 at 13-45-13 12.4 Wilcoxon Signed-Rank Test - Statistics LibreTexts.png"

[1] "Screenshot 2025-12-28 at 13-45-13 12.4 Wilcoxon Signed-Rank Test - Statistics LibreTexts.png"

The one-sample t-test

Suppose we have one sample of independent, identically distributed observations from some population, and we want to see whether we believe the population mean or median is some value, like 15. The standard test is the one-sample -test, where here we are pretending that we have a reason to know that the mean is greater than 15 if it is not 15:

x <- rnorm(n = 10, mean = 16, sd = 3)
x

 [1] 20.86560 13.76096 15.19321 13.90139 16.63971 18.12691 12.76501 18.37393
 [9] 16.01214 19.28764

t.test(x, mu = 15, alternative = "greater")

    One Sample t-test

data:  x
t = 1.7818, df = 9, p-value = 0.05423
alternative hypothesis: true mean is greater than 15
95 percent confidence interval:
 14.95704      Inf
sample estimates:
mean of x 
 16.49265 

In this case, the P-value is 0.0542, so we do not quite reject the null hypothesis that the population mean is 15: there is no evidence (at ) that the population mean is greater than 15. In this case, we have made a type II error, because the population mean is actually 16, and so the null hypothesis is actually wrong but we failed to reject it.

We use these data again later.

The theory behind the -test is that the population from which the sample is taken has a normal distribution. This is assessed in practice by looking at a histogram or a normal quantile plot of the data:

ggplot(tibble(x), aes(sample = x)) + stat_qq() + stat_qq_line()

With a small sample, it is hard to detect whether a deviation from normality like this indicates a non-normal population or is just randomness.¹ In this case, I actually generated my sample from a normal distribution, so I know the answer here is randomness.

There is another issue here, the Central Limit Theorem. This says, in words, that the sampling distribution of the sample mean from a large sample will be approximately normal, no matter what the population distribution is. How close the approximation is will depend on how non-normal the population is; if the population is very non-normal (for example, very skewed or has extreme outliers), it might take a very large sample for the approximation to be of any use.

Example: the chi-squared distribution is right-skewed, with one parameter, the degrees of freedom. As the degrees of freedom increases, the distribution becomes less skewed and more normal in shape.²

Consider the chi-squared distribution with 12 df:

tibble(x = seq(0, 30, 0.1)) %>% 
  mutate(y = dchisq(x, df = 12)) %>% 
  ggplot(aes(x = x, y = y)) + geom_line()

This is mildly skewed. Is a sample of size 20 from this distribution large enough to use the -test? We can simulate the sampling distribution of the sample mean, since we know what the population is, by drawing many (in this case 1000) samples from it and seeing now normal the simulated sample means look:

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(my_sample = list(rchisq(n = 20, df = 12))) %>% 
  mutate(my_mean = mean(my_sample)) %>% 
  ggplot(aes(sample = my_mean)) + stat_qq() + stat_qq_line()

This is a tiny bit skewed right (the very largest values are slightly too large and the very smallest ones not quite small enough, though the rest of the values hug the line), but I would consider this close enough to trust the -test.

Now consider the chi-squared distribution with 3 df, which is more skewed:

tibble(x = seq(0, 10, 0.1)) %>% 
  mutate(y = dchisq(x, df = 3)) %>% 
  ggplot(aes(x = x, y = y)) + geom_line()

How normal is the sampling distribution of the sampling mean now, again with a sample of size 20?

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(my_sample = list(rchisq(n = 20, df = 3))) %>% 
  mutate(my_mean = mean(my_sample)) %>% 
  ggplot(aes(sample = my_mean)) + stat_qq() + stat_qq_line()

This time, the normal quantile plot definitely strays from the line in a way that indicates a right-skewed non-normal sampling distribution of the sample mean. With this sample size, if the population is as skewed as a chi-squared distribution with 12 degrees of freedom, the -test is fine, but if it is as skewed as a chi-squared distribution with 3 degrees of freedom, the -test is at best questionable.

So, consideration of whether to use a -test has two parts: how normal the population is (answered by asking how normal your sample is), and how large the sample is. The larger the sample size is, the less the normality matters, but it is an awkward judgement call to assess whether the non-normality in the data distribution matters enough given the sample size.³

The sign test

If you have decided that your sample does not have close enough to a normal distribution (given the sample size), and therefore that you should not be using the -test, what do you do? Two standard options are the sign test and the signed-rank test, with the latter often being recommended over the former because of the former’s lack of power. These tests are both non-parametric, in that they do not depend on the data having (at least approximately) any specific distribution.

For the sign test, you count how many of your observations are above and below the null median. Here we use the same data as we used for the -test:

tibble(x) %>% 
  count(x > 15)

# A tibble: 2 × 2
  `x > 15`     n
  <lgl>    <int>
1 FALSE        3
2 TRUE         7

The number of values (say) above the null median is the test statistic. If the null hypothesis is true, each value is independently either above or below the null median with probability 0.5, and thus the test statistic has a binomial distribution with equal to the sample size and . Hence the P-value for an upper-tailed test is

sum(dbinom(7:10, size = 10, prob = 0.5))

[1] 0.171875

The split of 7 values above 15 and 3 below is still fairly close⁴ to 50–50, and so the P-value is large, much larger than for the -test.

The sign test does not use the data very efficiently: it only counts whether each data value is above or below the hypothesized median. Thus, if you are in a position to use the -test that uses the actual data values, you should do so. However, it is completely assumption-free: as long as the observations really are independent, it does not matter at all what the population distribution looks like.

The signed rank test

The signed-rank test occupies a kind of middle ground between the sign test and the -test.

Here’s how it works for our data, testing for a median of 15, against an upper-tailed alternative:

tibble(x) %>% 
  mutate(diff = x - 15) %>% 
  mutate(abs_diff = abs(diff)) %>% 
  mutate(rk = rank(abs_diff)) -> d
d

# A tibble: 10 × 4
       x   diff abs_diff    rk
   <dbl>  <dbl>    <dbl> <dbl>
 1  20.9  5.87     5.87     10
 2  13.8 -1.24     1.24      4
 3  15.2  0.193    0.193     1
 4  13.9 -1.10     1.10      3
 5  16.6  1.64     1.64      5
 6  18.1  3.13     3.13      7
 7  12.8 -2.23     2.23      6
 8  18.4  3.37     3.37      8
 9  16.0  1.01     1.01      2
10  19.3  4.29     4.29      9

Subtract the hypothesized median from each data value, and then rank the differences from smallest to largest in terms of absolute value. The smallest difference in size is 0.193, which gets rank 1, and the largest in size is , which gets rank 10. One of the negative differences is , which is the fifth largest in size (has rank 6).

The next stage is to sum up the ranks separately for the positive and negative differences:

d %>% group_by(diff > 0) %>% 
  summarize(sum = sum(rk))

# A tibble: 2 × 2
  `diff > 0`   sum
  <lgl>      <dbl>
1 FALSE         13
2 TRUE          42

There are only three negative differences, so their ranks add up to only 13, compared to a sum of 42 for the positive differences.⁵ For an upper-tailed test, the test statistic is the sum of the positive differences, which, if large enough, will lead to rejection of the null hypothesis.

Is 42 large enough to reject the null with?

wilcox.test(x, mu = 15, alternative = "greater")

    Wilcoxon signed rank exact test

data:  x
V = 42, p-value = 0.08008
alternative hypothesis: true location is greater than 15

The P-value is 0.0801, not small enough to reject the null median of 15 in favour of a larger value. It is a little bigger than for the -test, but smaller than for the sign test.

A historical note: the name usually attached to the signed-rank test is Frank Wilcoxon. He worked out the null distribution of the signed rank statistic (an exercise in combinatorics).

The R function is a bit of a confusing misnomer, because there was also a statistician called Walter Francis Willcox, who had nothing to do with this test.

Assessing the signed rank test

The signed-rank test seemed to behave well enough in our example, with actually normal data. But the point of mentioning the test is as something to use when the data are not normal.

So let’s take some samples from our skewed chi-squared distribution with 3 df.

This distribution has this median:

med <- qchisq(0.5, 3)
med

[1] 2.365974

and away we go. I’ll do 10,000 simulations this time:

tibble(sim=1:10000) %>% 
  rowwise() %>% 
  mutate(my_sample = list(rchisq(10, 3))) %>% 
  mutate(my_test = list(wilcox.test(my_sample, mu = med, alternative = "greater"))) %>% 
  mutate(my_p = my_test$p.value) -> dd
dd

# A tibble: 10,000 × 4
# Rowwise: 
     sim my_sample  my_test   my_p
   <int> <list>     <list>   <dbl>
 1     1 <dbl [10]> <htest> 0.862 
 2     2 <dbl [10]> <htest> 0.722 
 3     3 <dbl [10]> <htest> 0.5   
 4     4 <dbl [10]> <htest> 0.615 
 5     5 <dbl [10]> <htest> 0.0322
 6     6 <dbl [10]> <htest> 0.947 
 7     7 <dbl [10]> <htest> 0.0322
 8     8 <dbl [10]> <htest> 0.0244
 9     9 <dbl [10]> <htest> 0.0967
10    10 <dbl [10]> <htest> 0.539 
# ℹ 9,990 more rows

Since the null hypothesis is true, the P-values should have a uniform distribution:

ggplot(dd, aes(x = my_p)) + geom_histogram(bins = 12)

That doesn’t look very uniform, but rather skewed to the right, with too many low values, so that the test rejects more often than it should:⁶

dd %>% count(my_p <= 0.05)

# A tibble: 2 × 2
# Rowwise: 
  `my_p <= 0.05`     n
  <lgl>          <int>
1 FALSE           9194
2 TRUE             806

A supposed test at actually has a probability near 0.08 of making a type I error. (That’s why I did 10,000 simulations, in the hopes of eliminating sampling variability as a reason for it being different than 0.05.)

To investigate what happened, let’s look at one random sample and see whether we can reason it out:⁷

set.seed(457297)

x <- rchisq(10, 3)
x

 [1] 5.6369920 7.8136036 2.8290842 8.2463800 0.6228536 0.8004611 2.2627925
 [8] 6.3275717 1.2881783 0.6634575

and go through the calculations for the signed rank statistic again:

tibble(x) %>% 
  mutate(diff = x - med) %>% 
  mutate(abs_diff = abs(diff)) %>% 
  mutate(rk = rank(abs_diff)) -> d
d

# A tibble: 10 × 4
       x   diff abs_diff    rk
   <dbl>  <dbl>    <dbl> <dbl>
 1 5.64   3.27     3.27      7
 2 7.81   5.45     5.45      9
 3 2.83   0.463    0.463     2
 4 8.25   5.88     5.88     10
 5 0.623 -1.74     1.74      6
 6 0.800 -1.57     1.57      4
 7 2.26  -0.103    0.103     1
 8 6.33   3.96     3.96      8
 9 1.29  -1.08     1.08      3
10 0.663 -1.70     1.70      5

There are five positive and five negative differences, exactly as we would expect. But the positive differences are the four largest ones in size, so that the sum of the ranks for the positive differences is quite a bit larger than the sum of the ranks for the negative differences: 36 as against 19:

d %>% 
  group_by(diff > 0) %>% 
  summarize(sum = sum(rk))

# A tibble: 2 × 2
  `diff > 0`   sum
  <lgl>      <dbl>
1 FALSE         19
2 TRUE          36

This seems like a bit of a difference in rank sums, given that the null hypothesis is actually true.

Why did this happen, and why might it happen again? The population distribution is skewed to the right, so that there will occasionally be sample values much larger than the null median (even if that median is correct). There can not be sample values much smaller than the null median, because the distribution is bunched up at the bottom. That means that the positive differences will tend to be the largest ones in size, and hence the test statistic will tend to be bigger, and the P-value smaller, than ought to be the case.

Conclusions

The usual get-out for the above is to say that the signed-rank test only applies to symmetric distributions. Except that, one of the principal ways that the -test can fail is that the population distribution is skewed, and what we are then saying is that in that situation, we cannot use the signed-rank test either. Really, the only situation in which the signed-rank test has any value is when the population is symmetric with long tails or outliers, which seems to me a small fraction of the times when you would not want to use a -test.

So, the official recommendation is:

• when the population distribution seems normal enough (given the sample size), use the -test
• when the population distribution is not normal enough but is apparently symmetric, use the signed-rank test
• otherwise, use the sign test.

The second of those seems a bit unlikely (or unlikely to be sure enough about in practice), so that when I teach this stuff, it’s the -test or the sign test. As I have explained, the signed-rank test is only very rarely useful, and therefore, I contend, it is a waste of time to learn about it.

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Introduction

To see what might happen when a process is repeated many times, we can calculate. Or we can physically re-run the process many times, and count up the results: simulation of the process.

This can be applied to estimating probabilities, obtaining bootstrap distributions (for example when assessing normality), or estimating the power or size of tests.

I want my simulations here to be reproducible, so I will set the random number seed first:

set.seed(457299)

Tossing a coin

Imagine we toss a fair coin 10 times. How likely are we to get 8 or more heads? If you remember the binomial distribution, you can work it out. But if you don’t? Make a virtual coin, toss it 10 times, count the number of heads, repeat many times, see how many of those are 8 or greater.

Let’s set up our virtual coin first:

coin <- c("H", "T")

and, since getting a head on one toss doesn’t prevent a head on others, ten coin tosses would be a sample of size 10 with replacement from this coin:

sample(coin, 10, replace = TRUE)

 [1] "H" "T" "H" "H" "H" "H" "H" "H" "T" "T"

Seven heads this time.

I have a mechanism I use for “tidy simulation”:

• set up a dataframe with a column called sim to label the simulations
• work rowwise
• for each sim, do one copy of the thing you’ll be doing many times (in this case, simulating 10 coin tosses)
• calculate whatever you want to calculate for each sim
• summarize the results

For this problem, the code looks like this:

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(my_sample = list(sample(coin, 10, replace = TRUE))) %>% 
  mutate(heads = sum(my_sample == "H")) %>% 
  count(heads >= 8)

It is probably a good idea to run this one line at a time (to see what it does, and later as you develop your own).

In this case, 54 of the 1000 simulated sets of 10 coin tosses gave at least 8 heads, so our estimate of the probability of getting 8 or more heads in 10 tosses of a fair coin is 0.054.

Some notes about the code:

• I am using 1000 simulations as my “many” repeats of tossing a coin 10 times. A larger number would give a more accurate answer, but would take longer to run.¹
• working rowwise allows us to treat each row of the dataframe we are building as an independent entity. This makes the coding in the two mutates that follow much easier to follow, because our mental model only has to work one row at a time.²
• my_sample behaves like one sample of 10 coin tosses, though in fact it is a whole column of samples of 10 coin tosses. It is a vector of length 10, so to get it into one cell of our dataframe, we wrap it in list, making the whole column a list-column.
• Once again thinking of my_sample as a single sample, we then count the number of heads in it. I could use count, or table, but I don’t want to get caught by samples with no heads or no tails. This way counts 1 for each H in the sample, then adds up the counts.³
• Finally, count up the number of simulated sets of 10 coin tosses that had 8 or more heads. count accepts a logical condition as well as a column. (Behind the scenes it constructs a column of TRUE and FALSE first, and then counts that.)

In this case, we know the right answer:⁴

pbinom(7, 10, 0.5, lower.tail = FALSE)

[1] 0.0546875

Our simulation came out very close to this.

Aside:⁵ we can work out how accurate our simulation might be by noting that our 1000 simulations are also like Bernoulli trials: each one gives us 8 or more heads or it doesn’t, with unknown probability that is precisely the thing that we are trying to estimate. Thus:

binom.test(54, 1000)

    Exact binomial test

data:  54 and 1000
number of successes = 54, number of trials = 1000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.04082335 0.06987401
sample estimates:
probability of success 
                 0.054 

tells us, with 95% confidence, that the probability of 8 or more heads is between 0.041 and 0.070. To nail it down more precisely, use more than 1000 simulations.

How long is my longest suit?

In the game of bridge, each player, in two partnerships of 2, receives a hand of 13 cards randomly dealt from the usual deck of 52 cards. There is then an “auction” in which the two partnerships compete for the right to name the trump suit and play the hand. The bids in this auction are an undertaking to win a certain number of the 13 tricks with the named suit as trumps.⁶ Your partner cannot see your cards, and so in the bidding you have to share information about the strength and suit distribution of your hand using standard methods⁷ (you are not allowed to deceive your opponents), so that as a partnership you can decide how many tricks you can win between you.

One of the considerations in the bidding is the length of your longest suit, that is, the suit you hold the most cards in. The longest suit might have only 4 cards (eg. if you have 4 spades and 3 of each of the other suits), but if you are lucky⁸ you might be dealt a hand with 13 cards all of the same suit and have a longest suit of 13 cards. Evidently something in between those is more likely, but how likely?

For a simulation, we need to set up a deck of cards and select 13 cards from it without replacement (since you can’t draw the same card twice in the same hand). The only thing that matters here is the suits, so we’ll set up a deck with only suits and no denominations like Ace or King. (This will make the sampling without replacement look a bit odd.)

deck <- c(rep("S", 13), rep("H", 13),
          rep("D", 13), rep("C", 13))
deck

 [1] "S" "S" "S" "S" "S" "S" "S" "S" "S" "S" "S" "S" "S" "H" "H" "H" "H" "H" "H"
[20] "H" "H" "H" "H" "H" "H" "H" "D" "D" "D" "D" "D" "D" "D" "D" "D" "D" "D" "D"
[39] "D" "C" "C" "C" "C" "C" "C" "C" "C" "C" "C" "C" "C" "C"

and deal ourselves a hand of 13 cards thus:

hand <- sample(deck, 13, replace = FALSE)
hand

 [1] "D" "C" "C" "D" "S" "H" "S" "S" "D" "S" "H" "H" "H"

(note, for example, that the four Hearts in this hand are actually four different ones of the thirteen H in deck, since we are sampling without replacement. I could have labelled them by which Heart they were, but that would have made counting them more difficult.)

Then count the number of cards in each suit:

tab <- table(hand)
tab

hand
C D H S 
2 3 4 4 

This time the longest suit has four cards:

max(tab)

[1] 4

Using table is safe here, because we don’t care whether there are any suits with no cards in the hand, only about the greatest number of cards in any suit that we have cards in.⁹

All of that leads us to this:

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(hand = list(sample(deck, 13, replace = FALSE))) %>% 
  mutate(suits = list(table(hand))) %>% 
  mutate(longest = max(suits)) %>% 
  count(longest)

Note: the hands, and the tables of how many cards a hand has in each suit, are more than single numbers, so they need to be wrapped in list.

The most likely longest suit has 5 cards in it, a bit less than half the time. According to this, a longest suit of 8 cards happens about once in 500 hands, and longer longest suits are even less likely. (To estimate these small probabilities accurately, you need a lot of simulations, like, way more than 1000.)

Aside: the standard way of assessing hand strength is via high-card points: 4 for an ace, 3 for a king, 2 for a queen and one for a jack. All the other cards count zero. To simulate the number of points you might get in a hand, build a deck with the points for each card. There are four cards of each rank, and nine ranks that are worth no points:

deck <- c(rep(4,4), rep(3,4),
          rep(2,4), rep(1,4), rep(0, 36))

The simulation process after that is a lot like before:

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(hand = list(sample(deck, 13, replace = FALSE))) %>% 
  mutate(points = sum(hand)) -> d
d

I stopped it there, partly to show what the dataframe looks like at this point (a hand of 13 point values, and a points total that is the sum of these) and partly because I wanted to do about three things with this, and it made sense to save what we have done thus far.

First, a bar chart of how likely each number of points is:

ggplot(d, aes(x = points)) + geom_bar()

If you do more simulations, you can check whether the shape is indeed smooth (I’m guessing it is). The average number of points is 10 (there are 40 points in the deck and yours is one of four hands) and the distribution is right-skewed because it is possible, though rather unlikely, to get over 20 points.

In most bidding systems, having 13 points justifies opening the bidding (making the first bid in the auction if everyone has passed on their turn before you). How likely is that?

d %>% count(points >= 13)

Only about a quarter of the time.

Having 20 or more points qualifies your hand for an opening bid at the 2-level.¹⁰ How likely is that?

d %>% count(points >= 20)

A bit of a rarity, less than a 2% shot.

Bootstrapping a sampling distribution

To return to the messy world of actual applied statistics: there are a lot of procedures based on an assumption of the right things having normal distributions.¹¹ One of the commonest questions is whether we should be using the normal-theory procedure or something else (non-parametric, maybe). Let’s take an example. The data here are information about Toronto Blue Jays baseball games from the early part of the 2015 season:

my_url <- "http://ritsokiguess.site/datafiles/jays15-home.csv"
jays <- read_csv(my_url)

Rows: 25 Columns: 21
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (12): date, box, team, opp, result, wl, gb, winner, loser, save, Daynig...
dbl   (7): row, game, runs, Oppruns, innings, position, attendance
lgl   (1): venue
time  (1): game time

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

jays

There is a lot of information here, but we’re going to focus on the attendances over near the right side, and in particular, we’re interested in the mean attendance over all games of which these are a sample (“early-season Blue Jays games in the years between 2010 and 2019”, or something like that). There are, of course, lots of reasons that attendances might vary (opposition, weather, weekend vs. weekday, etc.) that we are going to completely ignore here.

The normal¹² way to estimate a population mean is to use the confidence interval based on the one-sample -test, but before we jump into that, we should look at a graph of the attendances:

ggplot(jays, aes(x = attendance)) + geom_histogram(bins = 6)

Well, that doesn’t look much like a normal distribution. It’s very much skewed to the right. There seem to be two¹³ schools of thought as to what we should do now:

• we have a large enough sample () so that we should get enough help from the central limit theorem (also expressed as “the -test is robust to non-normality”) and therefore the -procedure should be fine.
• this distribution is a long way from being normal, so there is no way we should use a -procedure, instead using a sign test or signed-rank test,¹⁴ inverted to get a confidence interval for the median attendance.

Both of these have an air of handwavery about them. How do we decide between them? Well, let’s think about this a little more carefully. When it comes to getting confidence limits, it all depends on the sampling distribution of the sample mean. If that is close enough to normal, the -interval is good. But this comes from repeated sampling. You conceptualize it by imagining taking lots of samples from the same population, working out the mean of each sample, and making something like a histogram or normal quantile plot of those. But but — we only have the one sample we have. How to think about possible sample means we might get?

A way around this is to use the bootstrap. The idea is to think of the sample we have as a population (resembling, we hope, the population we want to make inferences about of “all possible attendances”) and to take samples from our sample(!) of the same size as the sample we had. If we do this the obvious way (without replacement), we’ll get back the original sample we had, every time. So what we do instead is to sample from our sample, but with replacement so as to get a different set of values each time, with some values missing and some values repeated. Like this:

s <- sample(jays$attendance, replace = TRUE)
sort(s)

 [1] 15062 15062 15086 15086 15086 15168 16402 17264 17276 17276 18581 19217
[13] 21519 21519 21519 21519 21519 29306 29306 33086 34743 37929 42419 42917
[25] 44794

Sorting the sample reveals that the first two values and the next three are repeats, so there must be some values from the original sample that are missing. (This is the only reason I sorted them.)

The original data had a mean of

jays %>% summarise(mean_att = mean(attendance))

but the bootstrap sample has a mean of

mean(s)

[1] 23946.44

different; if we were to take more bootstrap samples, and find the mean of each one, we would get a sense of the sampling distribution of the sample mean. That is to say, we simulate the bootstrapped sampling distribution of the sample mean. Given what we’ve seen in the other simulations, the structure of the code below ought to come as no surprise:

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(s = list(sample(jays$attendance, replace = TRUE))) %>% 
  mutate(m = mean(s)) -> d
d

In words, set up the 1000 simulations and work rowwise as before, then take (for each row) a bootstrap sample of the attendances, and then take the mean of it. I’ve saved the resulting dataframe so that we can look at it and then do something else with it. The column s containing the samples is a list-column again.

Our question was whether this bootstrapped sampling distribution of the sample mean looked like a normal distribution. To see that, a normal quantile plot is the thing:

ggplot(d, aes(sample = m)) + stat_qq() + stat_qq_line()

That is very close to a normal distribution, and so in fact the -procedure really is fine and the first school of thought is correct (and now we have evidence, no hand-waving required):

t.test(jays$attendance)

    One Sample t-test

data:  jays$attendance
t = 11.389, df = 24, p-value = 3.661e-11
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 20526.82 29613.50
sample estimates:
mean of x 
 25070.16 

A 95% confidence interval for the mean attendance goes from 20500 to 29600.

Another way to go is to use the bootstrapped sampling distribution directly, entirely bypassing all the normal theory, and just take the middle 95% of it:

d %>% ungroup() %>% summarize(ci = quantile(m, c(0.025, 0.975)))

Warning: Returning more (or less) than 1 row per `summarise()` group was deprecated in
dplyr 1.1.0.
ℹ Please use `reframe()` instead.
ℹ When switching from `summarise()` to `reframe()`, remember that `reframe()`
  always returns an ungrouped data frame and adjust accordingly.

21000 to 29700, not that different (given the large amount of variability) from the -interval. There are better ways to get the interval rather than using sample quantiles; see for example here. But this will do for now.

The ungroup in the code is there because the dataframe d is still rowwise: everything we do with d will still be done one row at a time. But now we want to work on the whole column m, so we have to undo the rowwise first. rowwise is a special case of group_by (a sort of group-by-rows), so you undo rowwise in the same way that you undo group_by.

Power by simulation

power.t.test(n = 20, delta = 110 - 100, sd = 30, type = "one.sample", alternative = "one.sided")

     One-sample t test power calculation 

              n = 20
          delta = 10
             sd = 30
      sig.level = 0.05
          power = 0.4178514
    alternative = one.sided

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(20, 110, 30))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

power.t.test(power = 0.80, delta = 110 - 100, sd = 30, type = "one.sample", alternative = "one.sided")

     One-sample t test power calculation 

              n = 57.02048
          delta = 10
             sd = 30
      sig.level = 0.05
          power = 0.8
    alternative = one.sided

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(40, 110, 30))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

binom.test(6612, 10000)

    Exact binomial test

data:  6612 and 10000
number of successes = 6612, number of trials = 10000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.6518275 0.6704785
sample estimates:
probability of success 
                0.6612 

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(60, 110, 30))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

binom.test(8152, 10000)

    Exact binomial test

data:  8152 and 10000
number of successes = 8152, number of trials = 10000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.8074513 0.8227647
sample estimates:
probability of success 
                0.8152 

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(60, 110, 30))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

binom.test(811, 1000)

    Exact binomial test

data:  811 and 1000
number of successes = 811, number of trials = 1000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.7853323 0.8348215
sample estimates:
probability of success 
                 0.811 

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(20, 100, 30))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

binom.test(513, 10000)

    Exact binomial test

data:  513 and 10000
number of successes = 513, number of trials = 10000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.04705735 0.05580631
sample estimates:
probability of success 
                0.0513 

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rexp(20, 1/100))) %>% 
  mutate(t_test = list(t.test(sample, mu = 100, alternative = "greater"))) %>% 
  mutate(p_value = t_test$p.value) %>% 
  count(p_value <= 0.05)

binom.test(198, 10000)

    Exact binomial test

data:  198 and 10000
number of successes = 198, number of trials = 10000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.01716007 0.02272454
sample estimates:
probability of success 
                0.0198 

tibble(sim = 1:10000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(20, 100, 30))) %>% 
  mutate(sample_max = max(sample)) %>% 
  ungroup() %>% 
  summarize(pp = quantile(sample_max, 0.95))

tibble(sim = 1:1000) %>% 
  rowwise() %>% 
  mutate(sample = list(rnorm(20, 110, 30))) %>% 
  mutate(sample_max = max(sample)) %>% 
  ungroup() %>% 
  count(sample_max >= 184)

Final remarks

There is a lot of repetitiousness here. It would almost certainly be better to abstract the ideas of the simulation away into a function (that might have inputs the true parameter(s), the null parameter, the test and the population distribution), but one of the things I wanted to get across was that these all work the same way with a few small changes, which doesn’t come across quite so clearly by changing inputs to a function.

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(tmaptools)
library(leaflet)

Introduction

So I had a dataframe today, in which I wanted to make some small corrections. Specifically, I had this one:

my_url <- "http://ritsokiguess.site/datafiles/wisconsin.txt"
wisc <- read_table(my_url)
wisc %>% select(location)

These are mostly, but not all, cities in Wisconsin, and I want to draw them on a map. To do that, though, I need to affix their states to them, and I thought a good starting point was to start by pretending that they were all in Wisconsin, and then correct the ones that aren’t:

wisc %>% select(location) %>% 
  mutate(state = "WI") -> wisc
wisc

The last three cities are in the wrong state: Dubuque is in Iowa (IA), St. Paul in Minnesota (MN), and Chicago is in Illinois (IL). I know how to fix this in base R: I write something like

wisc$state[12] <- "IL"

but how do you do this the Tidyverse way?

A better way

The first step is to make a small dataframe with the cities that need to be corrected, and the states they are actually in:

corrections <- tribble(
  ~location, ~state,
  "Dubuque", "IA",
  "St.Paul", "MN",
  "Chicago", "IL"
)
corrections

Note that the columns of this dataframe have the same names as the ones in the original dataframe wisc.

So, I was thinking, this is a lookup table (of a sort), and so joining this to wisc might yield something helpful. We want to look up locations and not match states, since we want to have these three cities have their correct state as a possibility. So what does this do?

wisc %>% 
  left_join(corrections, by = "location")

Now, we have two states for each city. The first one is always Wisconsin, and the second one is usually missing, but where the state in state.y has a value, that is the true state of the city. So, the thought process is that the actual state should be:

• if state.y is not missing, use that
• else, use the value in state.x.

I had an idea that there was a function that would do exactly this, only I couldn’t remember its name, so I couldn’t really search for it. My first thought was na_if. What this does is every time it sees a certain value, it replaces it with NA. This, though, is the opposite way from what I wanted. So I looked at the See Also, and saw replace_na. This replaces NAs with a given value. Not quite right, but closer.

In the See Also for replace_na, I saw one more thing: coalesce, “replace NAs with values from other vectors”. Was that what I was thinking of? It was. The way it works is that you feed it several vectors, and the first one that is not missing gives its value to the result. Hence, what I needed was this:

wisc %>% 
  left_join(corrections, by = "location") %>% 
  mutate(state=coalesce(state.y, state.x))

Where state.y has a value, it is used; if it’s missing, the value in state.x is used instead.

The best way

I was quite pleased with myself for coming up with this, but I had missed the actual best way of doing this. In SQL, there is UPDATE, and what that does is to take a table of keys to look up and some new values for other columns to replace the ones in the original table. Because dplyr has a lot of things in common with SQL, it is perhaps no surprise that there is a rows_update, and for this job it is as simple as this:

wisc %>% 
  rows_update(corrections) -> wisc

Matching, by = "location"

wisc

The values to look up (the “keys”) are by default in the first column, which is where they are in corrections. If they had not been, I would have used a by in the same way as with a join.

Mind. Blown. (Well, my mind was, anyway.)

Geocoding

I said I wanted to draw a map with these cities on it. For that, I need to look up the longitude and latitude of these places, and for that, I need to glue the state onto the name of each city, to make sure I don’t look up the wrong one. It is perhaps easy to forget that unite is the cleanest way of doing this, particularly if you don’t want the individual columns any more:

wisc %>% unite(where, c(location, state), sep = " ") -> wisc
wisc

The function geocode_OSM from tmaptools will find the longitude and latitude of a place. It expects one place as input, not a vector of placenames, so we will work rowwise to geocode one at a time. (Using map from purrr is also an option.) The geocoder returns a list, which contains, buried a little deeply, the longitudes and latitudes:

wisc %>% 
  rowwise() %>% 
  mutate(ll = list(geocode_OSM(where))) -> wisc
wisc

The column ll is a list-column, and the usual way to handle these is to unnest, but that isn’t quite right here:

wisc %>% unnest(ll)

Unnesting a list of three things produces three rows for each city. It would make more sense to have the unnesting go to the right and produce a new column for each thing in the list. The new tidyr has a variant called unnest_wider that does this:

wisc %>% 
  unnest_wider(ll)

The longitudes and latitudes we want are still hidden in a list-column, the one called coords, so with luck, if we unnest that wider as well, we should be in business:

wisc %>% 
  unnest_wider(ll) %>% 
  unnest_wider(coords) -> wisc
wisc

And now we are. x contains the longitudes (negative for degrees west), and y the latitudes (positive for degrees north).

Making a map with these on them

The most enjoyable way to make a map in R is to use the leaflet package. Making a map is a three-step process:

• leaflet() with the name of the dataframe
• addTiles() to get map tiles to draw the map with
• add some kind of markers to show where the points are. I use circle markers here; there are also markers (from addMarkers) that look like Google map pins. Here also you associate the longs and lats with the columns they are in in your dataframe:

leaflet(data = wisc) %>% 
  addTiles() %>% 
  addCircleMarkers(lng = ~x, lat = ~y) 

The nice thing about Leaflet maps is that you can zoom, pan and generally move about in them. For example, you can zoom in to find out which city each circle represents.

Introduction

Do you follow @londonmapbot on Twitter? You should. Every so often a satellite photo is posted of somewhere in London (the one in England), with the implied invitation to guess where it is. Along with the tweet is a link to openstreetmap, and if you click on it, it gives you a map of where the photo is, so you can see whether your guess was right. Or, if you’re me, you look at the latitude and longitude in the link, and figure out roughly where in the city it is. My strategy is to note that Oxford Circus, in the centre of London, is at about 51.5 north and 0.15 west, and work from there.¹

Matt Dray, who is behind @londonmapbot, selects random points in a rectangle that goes as far in each compass direction as the M25 goes. (This motorway surrounds London in something like a circle, and is often taken as a definition of what is considered to be London; if outside, not in London. There is a surprising amount of countryside inside the M25.)

London has the advantage of being roughly a rectangle aligned north-south and east-west, and is therefore easy to sample from. I have been thinking about doing something similar for my home city Toronto, but I ran into an immediate problem:

Toronto is not nicely aligned north-south and east-west, and so if you sample from a rectangle enclosing it, this is what will happen:

You get some points inside the city, but you will also get a number of points in Vaughan or Mississauga or Pickering or Lake Ontario! How to eliminate the ones I don’t want?

Sampling from a region

Let’s load some packages:

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(leaflet)
library(sp)

I had this vague idea that it would be possible to decide if a sampled point was inside a polygon or not. So I figured I would start by defining the boundary of Toronto as a collection of straight lines joining points, at least approximately. The northern boundary of Toronto is Steeles Avenue, all the way across, and that is a straight line, but the southern boundary is Lake Ontario, and the western and eastern boundaries are a mixture of streets and rivers, so I tried to pick points which, when joined by straight lines, enclosed all of Toronto without too much extra. This is what I came up with:

boundary <- tribble(
  ~where, ~lat, ~long,
"Steeles @ 427", 43.75, -79.639,
"Steeles @ Pickering Townline", 43.855, -79.17,
"Twyn Rivers @ Rouge River", 43.815, -79.15,
"Rouge Beach", 43.795, -79.115,
"Tommy Thompson Park", 43.61, -79.33,
"Gibraltar Point", 43.61, -79.39,
"Sunnyside Beach", 43.635, -79.45,
"Cliff Lumsden Park", 43.59, -79.50,
"Marie Curtis Park", 43.58, -79.54,
"Rathburn @ Mill", 43.645, -79.59,
"Etobicoke Creek @ Eglinton", 43.645, -79.61,
"Eglinton @ Renforth", 43.665, -79.59,
"Steeles @ 427", 43.75, -79.639,
)
boundary

I kind of had the idea that you could determine whether a point was inside a polygon or not. The idea turns out to be this: you draw a line to the right from your point; if it crosses the boundary of the polygon an odd number of times, it’s inside, and if an even number of times, it’s outside. So is there something like this in R? Yes: this function in the sp package.²

So now I could generate some points in the enclosing rectangle and see whether they were inside or outside the city, like this:

set.seed(457299)
n_point <- 20
tibble(lat = runif(n_point, min(boundary$lat), max(boundary$lat)),
       long = runif(n_point, min(boundary$long), max(boundary$long))) -> d
d %>% mutate(inside = point.in.polygon(d$long, d$lat, boundary$long, boundary$lat)) %>% 
  mutate(colour = ifelse(inside == 1, "blue", "red")) -> d
d

The function point.in.polygon returns a 1 if the point is inside the polygon (city boundary) and a 0 if outside.³

I added a column colour to plot the inside and outside points in different colours on a map, which we do next. The leaflet package is much the easiest way to do this:

leaflet(d) %>% 
  addTiles() %>% 
  addCircleMarkers(color = d$colour) %>% 
    addPolygons(boundary$long, boundary$lat)

Assuming "long" and "lat" are longitude and latitude, respectively

The polygons come from a different dataframe, so I need to specify that in addPolygons. Leaflet is clever enough to figure out which is longitude and which latitude (there are several possibilities it will understand).

This one seems to have classified the points more or less correctly. The bottom left red circle is just in the lake, though it looks as if one of the three rightmost blue circles is in the lake also. Oops. The way to test this is to generate several sets of random points, test the ones near the boundary, and if they were classified wrongly, tweak the boundary points and try again. The coastline around the Scarborough Bluffs is not as straight as I was hoping.

Mapbox

Matt Dray’s blog post gives a nice clear explanation of how to set up MapBox to return you a satellite image of a lat and long you feed it. What you need is a Mapbox API key. A good place to save this is in your .Renviron, and edit_r_environ from usethis is a good way to get at that. Then you use this key to construct a URL that will return you an image of that point.

Let’s grab one of those sampled points that actually is in Toronto:

d %>% filter(inside == 1) %>% slice(1) -> d1
d1

and then I get my API key and use it to make a URL for an image at this point:

mapbox_token <- Sys.getenv("MAPBOX_TOKEN")
url <- str_c("https://api.mapbox.com/styles/v1/mapbox/satellite-v9/static/",
             d1$long,
             ",",
             d1$lat,
             ",15,0/600x400?access_token=",
             mapbox_token)

I’m not showing you the actual URL, since it contains my key! The last-but-one line contains the zoom (15) and the size of the image (600 by 400). These are slightly more zoomed out and bigger than the values Matt uses. (I wanted to have a wider area to make it easier to guess.)

Then download this and save it somewhere:

where <- "img.png"
download.file(url, where)

and display it:

I don’t recognize that, so I’ll fire up leaflet again:

leaflet(d1) %>% 
  addTiles() %>% 
  addCircleMarkers() 

Assuming "long" and "lat" are longitude and latitude, respectively

It’s the bit of Toronto that’s almost in Mississauga. The boundary is Etobicoke Creek, at the bottom left of the image.

References

How to determine if point inside polygon

point.in.polygon function documentation

Matt Dray blog post on londonmapbot

Introduction

Some cars have a computer that records gas mileage since the last time the computer was reset. A driver is concerned that the computer on their car is not as accurate as it might be, so they keep an old-fashioned notebook and record the miles driven since the last fillup, and the amount of gas filled up, and use that to compute the miles per gallon. They also record what the car’s computer says the miles per gallon was.

Is there a systematic difference between the computer’s values and the driver’s? If so, which way does it go?

Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.6
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.1     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

The data

The driver’s notebook has small pages, so the data look like this:

Fillup     1    2    3    4    5
Computer 41.5 50.7 36.6 37.3 34.2
Driver   36.5 44.2 37.2 35.6 30.5
Fillup     6    7    8    9   10
Computer 45.0 48.0 43.2 47.7 42.2
Driver   40.5 40.0 41.0 42.8 39.2
Fillup    11   12   13   14   15
Computer 43.2 44.6 48.4 46.4 46.8
Driver   38.8 44.5 45.4 45.3 45.7
Fillup    16   17   18   19   20
Computer 39.2 37.3 43.5 44.3 43.3
Driver   34.2 35.2 39.8 44.9 47.5

This is not very close to tidy. There are three variables: the fillup number (identification), the computer’s miles-per-gallon value, and the driver’s. These should be in columns, not rows. Also, there are really four sets of rows, because of the way the data was recorded. How are we going to make this tidy?

Making it tidy

As ever, we take this one step at a time, building a pipeline as we go: we see what each step produces before figuring out what to do next.

The first thing is to read the data in; these are aligned columns, so read_table is the thing. Also, there are no column headers, so we have to say that as well:

my_url <- "gas-mileage.txt"
gas <- read_table(my_url, col_names = FALSE)

── Column specification ────────────────────────────────────────────────────────
cols(
  X1 = col_character(),
  X2 = col_double(),
  X3 = col_double(),
  X4 = col_double(),
  X5 = col_double(),
  X6 = col_double()
)

gas

gas %>% pivot_longer(X2:X6, names_to = "var_name", values_to = "var_value")

gas %>% pivot_longer(X2:X6, names_to = "var_name", values_to = "var_value") %>% 
  pivot_wider(names_from = X1, values_from = var_value)  

Warning: Values from `var_value` are not uniquely identified; output will contain
list-cols.
• Use `values_fn = list` to suppress this warning.
• Use `values_fn = {summary_fun}` to summarise duplicates.
• Use the following dplyr code to identify duplicates.
  {data} |>
  dplyr::summarise(n = dplyr::n(), .by = c(var_name, X1)) |>
  dplyr::filter(n > 1L)

gas %>% pivot_longer(X2:X6, names_to = "var_name", values_to = "var_value") %>% 
  spread(X1, var_value)