Garrick Aden-Buie

Remembering what I did this week with LLMs

Sat, 01 Feb 2025 21:41:16 GMT

The object impermanence of work

My least favorite meetings are status meetings. I love seeing the people I work with and hearing about what they’ve been up to, but I hate trying to remember what the hell I’ve been working on.

When I joined the Shiny team, we’d have a short daily status meeting and you’d think it’d be easy to remember what I did in the last 24 hours. But apparently, that’s not how I live my life. I’m sure I thought of things to say, but every daily update I felt like I’d say some variation of “I did a bunch of stuff and now I’m… doing more stuff”.

My perception of how I work is in short bursts of deep focus time punctuated by many short breaks to answer questions on Slack or GitHub issues or the Shiny Discord server or Posit Community or any of the many other places that people post questions and sometimes code.

Between the deep focus work, I usually maintain one thread of maintenance tasks, like babysitting a CI process, or making a small change to fix a bug, or turning screenshots into reproducible bug reports. And I also tend to have another more speculative thread going for learning something new or playing with a new idea.

I don’t recommend my way of working to others. Whole books have been written just to say “don’t work like this”! Software engineers rail against context switching, interruptions, and multitasking.

I compensate for the known perils of my work style in two ways. First, even if I do a lot of different things throughout the day, there’s generally only ever one task that I’m working on at any given moment. That task gets all the focus.

Second, I try to leave context behind in my work. If I’m working on a pull request, that means submitting the PR with enough description that I could forget that I even wrote the PR and most of what I’d need to pick it back up is there in the description or comments. If I had some code in front of me when I found the issue or was testing the PR, that code ends up in the description. If the context is recoverable, it’s at least not lost forever in the switching.

But these two strategies combine in a strange way, resulting in the object impermanence of work: if it’s not in front of me, it doesn’t exist.

I just don’t keep a running inventory of what I was working on or what I did; I know that I can find everything I need to know from the open GitHub issues or PRs. Which then are summarized in stand ups as… stuff.

What did I even do this week?

It turns out I can answer the question of what I did this week, as long as I’m allowed to run some R code.

Since my entire life is lived on GitHub, I can get a pretty good sense of what I’ve been working on by looking at the PRs and issues I opened and closed in a week. Even better, this information is readily obtained from GitHub with a search like author:gadenbuie created:>2025-01-17T00:00:00Z or author:gadenbuie closed:>2025-01-17T00:00:00Z. And I can create those search queries with R:

author <- "gadenbuie"
date <- Sys.Date() - 7
search_created <- sprintf("author:%s created:>%sT00:00:00Z", author, date)
search_closed <- sprintf("author:%s closed:>%sT00:00:00Z", author, date)

But of course I don’t want to have to actually click around on GitHub to remember what I was doing. What I want is some rectangular data that I can work on in R.

Enter the gh package, a package that lets you interact with the GitHub API. We can use gh to run the created and closed search queries and collect all of the results.

library(gh)

created <- gh("/search/issues", q = search_created, .limit = Inf)
closed <- gh("/search/issues", q = search_closed, .limit = Inf)

Why not do this in one single API request? Unfortunately, GitHub search doesn’t let you use OR for created: and closed: in the same query, but that’s okay, we can put the results together ourselves. Each result has an $items field that contains a list of issues/pull requests, each with a whole bunch of fields. I’ll use purrr to map over the issues and extract the fields I care about.

library(purrr)
library(dplyr, warn.conflicts = FALSE)

work_data <-
  c(created$items, closed$items) |>
    map(function(issue) {
      tibble(
        repository = sub(
          "https://api.github.com/repos/",
          "",
          issue$repository_url
        ),
        title = issue$title,
        created_at = issue$created_at,
        closed_at = issue$closed_at,
        url = issue$html_url,
        state = issue$state,
        is_issue = is.null(issue$pull_request),
        body = issue$body
      )
    }) |>
    list_rbind()

work_data

# A tibble: 83 × 8
   repository              title created_at url   state is_issue body  closed_at
                                        
 1 r-lib/systemfonts       Repo… 2025-01-2… http… open  TRUE     "A s…      
 2 rstudio/shiny           fix(… 2025-01-2… http… open  FALSE    "Thi…      
 3 rstudio/chromote        v0.4… 2025-01-2… http… open  FALSE    "For…      
 4 rstudio/chromote        Rele… 2025-01-2… http… open  TRUE     "Pre…      
 5 quarto-ext/bluesky-com… refa… 2025-01-2… http… open  FALSE    "Fix…      
 6 quarto-ext/bluesky-com… Add … 2025-01-2… http… open  TRUE     "The…      
 7 rstudio/bslib           Bran… 2025-01-2… http… clos… FALSE    "Fix… 2025-01-…
 8 rstudio/bslib           chor… 2025-01-2… http… clos… FALSE    "Mov… 2025-01-…
 9 rstudio/bslib           Bran… 2025-01-2… http… clos… TRUE     "The… 2025-01-…
10 quarto-dev/quarto-cli   All … 2025-01-2… http… open  TRUE     "###…      
# ℹ 73 more rows

Now that I have a nice rectangular summary of my work over the last week, I’ll organize it a bit to differentiate between my Posit/Shiny work and my personal open-source projects. I’ll also add an action column indicating the primary action this week.

work_data <-
  work_data |>
    mutate(
      personal = grepl("(^gadenbuie)|bluesky-comments", repository),
      action_at = coalesce(closed_at, created_at),
      action = recode(state, open = "opened"),
      action = paste(action, if_else(is_issue, "an issue", "a PR")),
    ) |>
    arrange(personal, repository, is_issue, action_at)

# Preview the work data
work_data |>
  select(repository, action, action_at, title)

# A tibble: 83 × 4
   repository                  action          action_at            title       
                                                            
 1 posit-dev/air               opened an issue 2025-01-17T19:11:42Z Try to brea…
 2 posit-dev/air               opened an issue 2025-01-17T20:11:24Z Empty or sh…
 3 posit-dev/air               closed an issue 2025-01-21T19:56:11Z Selectively…
 4 posit-dev/air               closed an issue 2025-01-21T19:56:11Z Selectively…
 5 posit-dev/brand-yml         closed a PR     2025-01-21T17:32:35Z docs: Add g…
 6 posit-dev/brand-yml         closed a PR     2025-01-21T17:32:35Z docs: Add g…
 7 posit-dev/brand-yml         closed a PR     2025-01-21T19:58:30Z docs: Use G…
 8 posit-dev/brand-yml         closed a PR     2025-01-21T19:58:30Z docs: Use G…
 9 posit-dev/py-shiny          opened an issue 2025-01-21T16:25:43Z `render.Dat…
10 posit-dev/py-shiny-validate closed a PR     2025-01-22T17:28:32Z fix: Disabl…
# ℹ 73 more rows

Rectangular data FTW! This is great, I have a nice table of all the things I’ve done this week. But it’s still a bit hard to read to make sense of it all. We’ve reached the “I did some stuff and it’s in this tibble” stage, which is admittedly a little better than “I did some stuff”.

What I’d love to do is give this to a robot assistant who could read it all for me and they could summarize my week, or at least get me started on a summary. Sounds like a great task for a large language model!

Hiring my executive robot assistant

Now we get to the fun part!¹ We’re going to use the newly released ellmer package to send my work summary to a large language model and ask it to summarize my week.

If you’re following along at home, here’s where you’ll need to pause and get yourself set up. First, install ellmer:

install.packages("ellmer")

Second, you’ll need access to an LLM. The coolest part of ellmer is that it supports a wide range of LLM providers, which are all listed in the Chatbots reference. To use your favorite LLM, you’ll need to set up an account and get an API key. But once you’ve completed that step, switching between LLMs is as easy as changing out the chat_*() function that you use to start the chat.

For this demo, I’ll use Anthropic with chat_claude(). The help page for chat_claude() includes instructions for setting up a developer account, which I’ve already followed.

library(ellmer)

chat <- chat_claude()

If you prefer OpenAI’s ChatGPT, you’d use chat_openai(), or if you have ollama running a local model you’d use chat_ollama().

Once you have a chat object, you can use the $chat() method to talk to the LLM. The answers are streamed back to your R session and the chat is cumulative, so you can keep “talking”.

chat$chat("Say hello three languages.")

Here's "hello" in three languages:

1. English: Hello
2. Spanish: Hola
3. French: Bonjour

chat$chat("Say goodbye in those same languages.")

Here's "goodbye" in those same languages:

1. English: Goodbye
2. Spanish: Adiós
3. French: Au revoir

Our final task is to give the LLM all of the information we’ve collected from my GitHub issues and then to convince it to summarize it all in a meaningful way. I can further refine this task into three distinct components:

Use a system prompt to vibe check the LLM. System prompts are a good place to store initial instructions that the LLM uses a sort of “light instruction set”, like a sort of zero-th prompt.

I’ll use the system prompt to convince the LLM that it’s my assistant.
Turn the work summary into something a little bit more readable that I can send to the LLM as part of the user prompt.
Write up a user prompt that gets the LLM to do what I want (most of the time).

System prompt

This is the system prompt that I settled on:

_prompt-system.md

You are an assistant who keeps track of the work I do each week. Your goal is to
faithfully and accurately report the work I do to my boss while making sure that
the impact of my work is clearly articulated. You are clear and concise and use
direct language, preferring to match the terminology and phrases that I use. You
respond directly without any preamble.

The goal is to try to help the LLM align with the vibe of the task:

It needs to summarize my work
Please don’t make things up
None of that “Sure, here’s your summary of…” fluff.

Work context

The next step is to get the work summary into a format that’s easier for LLMs to work with. There’s a good chance that I could just throw CSV or a JSON of the work_data table at it, but I think I’ll get better answers if I format it like text. Notice that I’ve already grouped projects together and I’ve tried to make sure relevant parts of the work data are close to each other. I’m going off vibes with this, but I expect I’ll get better summaries with a little pre-work than if I’d ask the LLM to make connections at great distance across the text.

For this task, I’ll turn to my favorite string templating package, epoxy. I’ve created a template that I’ll apply to each row of work_data:

_prompt-work-summary.md

---
engine: epoxy
---

<work-item>
Action: {action}
Title: {title}
Type: {ifelse(personal, 'Personal Work', 'Professional Work')}
Link: {url}

<work-description>
{body}
work-description>
work-item>

I can take work_data and pass it to epoxy::epoxy_use_file() to fill in the templating bits with the values from work_data, one row at a time.

library(epoxy)

work_data |>
  slice(15) |>
  epoxy_use_file("_prompt-work-summary.md", .engine)

Here’s an example of a single work summary using the 15th row.

<work-item>
Action: closed a PR
Title: chore: Add `done.` signal when package installation is completed
Type: Professional Work
Link: https://github.com/posit-dev/shinylive/pull/195

<work-description>
In #194 we went back to calling

```patch
- micropip.install(pkg_name)
+ micropip.install(req)
```

because otherwise we lose information like the requested package version.

I thought this broke installation of package extras because it appeared that `micropip.install()` doesn't actually install the packages listed in an extra. 

It turns out that extras aren't installed when the main package is already installed. In other words, `micropip.install("shiny[theme]")` will _not install_ `brand_yml` and `libsass` because `shiny` is already installed, but `micropip.install("httpx[socks]")` _will install_ `socksio` along with `httpx` because `httpx` isn't initially installed.

So in the end, this PR mostly just adjusts a few comments and improves the UX of the console message. As [pointed out on Discord](https://discord.com/channels/1109483223987277844/1318357786379161731/1318536291750641674) the messages say `Installing {req} ...` and the last message seems to hang. This PR adds `done` to each line when the package install finishes.
work-description>
work-item>

Notice that I’ve used and pseudo-tags to give the text a bit of structure.

Now we can put the work summary together into a single text string.

work_summary <-
  work_data |>
    epoxy_use_file("_prompt-work-summary.md", .collapse = "\n\n")

The user prompt

The last step is to give the LLM some instructions about what I want it to write. Again, I’ll use a prompt file for this task.

_prompt.md

Summarize my work over the last week for a team update in prose.
Focus more on pull requests; issues are less important and just provide context.
Write in prose with full paragraphs.
Focus on the user impact of the work first and secondarily on the internal impact of dev changes.

Summarize the work using markdown under two headings:
'## Professional Work' and '## Personal Work'.
Include direct links to any mentioned pull requests using markdown syntax.
Use the first-person and try to sound like me.

<work-summary>
{{ work_summary }}
work-summary>

For this prompt, I’ll use ellmer::interpolate_file(), which works a lot like epoxy_use_file() but comes built-in with ellmer. It reads in a file and interpolates the { expr } strings, which in this case will inject the work_summary into the prompt.

The final result

Finally, we’re ready to put it all together and find out if it works!

chat <- chat_claude(
  system_prompt = readLines("_prompt-system.md")
)

work_summarized <-
  chat$chat(interpolate_file("_prompt.md"), echo = FALSE)

# I'm caching the result so I don't rack up costs while writing
saveRDS(chat, "chat.rds")

I’ve set echo=FALSE so that I can store the result and show it to you with a big reveal, but by default the response streams right into your console.

Sending my whole work summary plus the prompt took about 23,297 tokens and Claude responded with 594 tokens. Using LLM Price Check, 25k input tokens and 500k output tokens, ends up costing about $0.08 per call to claude-3-sonnet or about $0.13 per call to gpt-4o.

Here’s the final output! ellmer has a really neat feature that you can turn any part of the conversation into markdown with contents_markdown(). I’ve used code like what’s listed below to embed the response here in this post.

chat$last_turn() |>
  contents_markdown()

My Work Summary

Professional Work

I made significant improvements across multiple projects this week. In bslib, I improved navbar compatibility for Bootstrap 5 and light/dark modes through PR #1145, which brought bslib in line with Bootstrap 5’s expectations around navbar background color control while adding appropriate controls for various scenarios. I also contributed to better code organization by moving navbar options to a dedicated file through PR #1173.

In chromote, I enhanced the developer experience by adding the chromote_info() helper function via PR #190 to provide useful system and configuration information. I also improved session management by setting user-specific crash dump directories in PR #194.

For shinylive and Shiny, I made several improvements to enhance stability and user experience. I fixed Windows path handling in shinylive through PR #47 and improved the insertion of nav items in Shiny with PR #4179.

I also worked on improving code formatting across our repositories by setting up automated R code formatting with air through PR #43 in the shiny-workflows repository.

Personal Work

I made substantial contributions to the bluesky-comments Quarto extension, implementing several key features and improvements:

Enhanced content label handling and UI through PR #29
Added support for images and GIFs in PR #40
Implemented nested comment depth control via PR #41
Improved header configuration options in PR #39

I also enhanced shrtcts to support better workflow integration in Positron through PR #26, adding shortcut_run() and shortcut_get() functions.

I have to admit, that’s not a bad summary² of my week!

What’s cool about this approach is that I could continue to chat with Claude about my week, asking follow up questions to have it further remind me about what I did.

Another idea I have now is to run this script in a weekly cron job, just before our larger team meeting, to make sure that I know what I did this week when I get there. Not only that, but this would make a great running summary that might come in handy for a performance review or promotion request.

Using local LLMS

Just to prove that it’s easy to switch models, I’m going to repeat the above process with a local model. This is the same code as above, but this time using ollama to run phi4, a new 14B parameter model from Microsoft.

Before I show you the results, I have to admit something: they’re … not stellar. I tried a few other local models, and phi4 generally did the best. A 14B parameter model isn’t the smallest model on the block, but it’s still pretty small. By comparison, Claude 3.5 Sonnet has roughly 175 billion parameters. The smaller parameter models have almost no chance of acing this task.

I also found that if I tried to summarize both professional and personal work in the same chat, the local models would get hung up on the personal work and would end up summarizing only that. After playing around with it for a bit, I decided to focus just on the professional work summary, where phi4 does okay.

professional_work <-
  work_data |>
    filter(!personal) |>
    epoxy_use_file("_prompt-work-summary.md", .collapse = "\n\n")

chat_llama <- chat_ollama(
  system_prompt = brio::read_file("_prompt-system.md"),
  model = "phi4",
  api_args = list(temperature = 0.2)
)

work_summarized_llama <-
  chat_llama$chat(
    interpolate_file(
      "_prompt.md",
      work_summary = professional_work
    ),
    echo = FALSE
  )

My Work Summary (phi4)

Summary of Recent Work

Professional Activities:

Shiny Core CI Snapshots Update:
- Updated snapshots due to Chrome v132’s deprecation of --headless=old mode.
- Chromote now uses the new --headless mode by default (see Chromote PR #189).
- Reviewed all updated snapshots; no behavior changes detected, except for potential issues with absolute positioning in panel layouts (027-absolutely-positioned-panels).
Shiny Workflows Enhancement:
- Introduced a new composite workflow rstudio/shiny-workflows/format-r-code to format R code using air.
- This workflow is part of the routine process, activated via inputs.format-r-code, which defaults to false.
- Users can enable it by setting format-r-code: true in their configuration.
- Additional inputs for this workflow include:
  - version: Specifies the version of air (default: latest).
  - check: When set to "true", checks if files change without committing (air format --check).
  - path: Defines paths for formatting, defaulting to all R files in the repository.
- Demonstrated usage in the bslib project (bslib PR #1167).
Shiny Core CI Snapshots Update (Duplicate Entry):
- Reiterated update due to Chrome v132 changes, with similar outcomes and considerations as previously noted.

Open Source Contributions:

Shiny PR #4952:
- Addressed a bug where shiny::runApp() would fail if the app directory was not writable.
- Implemented a fix that allows apps to run in read-only directories by copying files into a temporary location for execution.
Shiny Core CI Snapshots Update (Duplicate Entry):
- Repeated update details regarding Chrome v132 and Chromote’s headless mode transition, with no additional changes noted beyond the initial entry.

These activities reflect ongoing efforts to maintain and enhance the Shiny ecosystem, ensuring compatibility with new browser versions and improving developer workflows.

The local model takes quite a bit longer to provide a response, but I can run it on my M3 MacBook Air. The summarization quality is nowhere near what I got from Claude, either, but I didn’t need to pay for the query or send my data to a third-party. (I also didn’t spend much time on this, I probably could get better results with more prompt engineering.) At the very least, the local model is a great way to practice setting up the pipeline before I’m ready to pay for queries with Anthropic or others.

Final script

One last thing: here’s all of the code from the meandering blog post above in one place, with all of the prompts inlined. Amazingly, its less than 100 lines of code.

# Query issues and pull requests from this week -------------------------------
library(dplyr)
library(purrr)
library(gh)

author <- "gadenbuie"
start_date <- Sys.Date() - 7

search_created <- sprintf("author:%s created:>%sT00:00:00Z", author, start_date)
search_closed <- sprintf("author:%s closed:>%sT00:00:00Z", author, start_date)

created <- gh("/search/issues", q = search_created, .limit = Inf)
closed <- gh("/search/issues", q = search_closed, .limit = Inf)

work_data <-
  c(created$items, closed$items) |>
    map(function(issue) {
      tibble(
        repository = sub(
          "https://api.github.com/repos/",
          "",
          issue$repository_url
        ),
        title = issue$title,
        created_at = issue$created_at,
        closed_at = issue$closed_at,
        url = issue$html_url,
        state = issue$state,
        is_issue = is.null(issue$pull_request),
        body = issue$body
      )
    }) |>
    list_rbind() |>
      mutate(
        personal = grepl("(^gadenbuie)|bluesky-comments", repository),
        action_at = coalesce(closed_at, created_at),
        action = recode(state, open = "opened"),
        action = paste(action, if_else(is_issue, "an issue", "a PR")),
      ) |>
      arrange(personal, repository, is_issue, action_at)

# Prepare the work data summary -----------------------------------------------
library(epoxy)

work_summary <-
  epoxy(
    "
    Action: {action}
    Title: {title}
    Type: {ifelse(personal, 'Personal Work', 'Professional Work')}
    Link: {url}

    
    {body}
    
    ",
    .collapse = "\n\n"
  )

# Summarize with the LLM ------------------------------------------------------
library(ellmer)

chat <- chat_claude(
  system_prompt = epoxy(
    "You are an assistant who keeps track of the work I do each week. Your goal is to
    faithfully and accurately report the work I do to my boss while making sure that
    the impact of my work is clearly articulated. You are clear and concise and use
    direct language, preferring to match the terminology and phrases that I use. You
    respond directly without any preamble."
  )
)

chat$chat(
  interpolate(
    "Summarize my work over the last week for a team update in prose.
    Focus more on pull requests; issues are less important and just provide context.
    Write in prose with full paragraphs.
    Focus on the user impact of the work first and secondarily on the internal impact of dev changes.

    Summarize the work using markdown under two headings:
    '## Professional Work' and '## Personal Work'.
    Include direct links to any mentioned pull requests using markdown syntax.
    Use the first-person and try to sound like me.

    
    {{ work_summary }}
    "
  )
)

# To get the LLM response as markdown:
contents_markdown(chat$last_turn())

Footnotes

I know what you’re thinking and you’re totally right. The fun part was getting all this data in a nice tidy tibble! Think of all the things we could do with that tibble. But dplyr isn’t the new hotness and this blog needs to meet its buzz word quota, so we’re forced to do something extra.↩︎
Does the summary have links to the PRs? This one was hit-or-miss. Sometimes Claude includes these and sometimes they’re omitted. I’m sure a little more prompt massaging could improve the consistency.↩︎

It’s time to add bslib to your shinyapp snippet

Mon, 22 Apr 2024 04:00:00 GMT

Last week, I presented at ShinyConf 2024 about using bslib to make modern dashboards with Shiny. It was a fun talk to put together and present, and the conference itself was full of great talks showcasing the broad range of Shiny apps, developers and uses.

My talk focused on the bslib package, which was also the main theme of my first year on the Shiny team. bslib started out as a way to get the latest versions of Bootstrap into existing Shiny apps, but over the last couple of years the package has grown to include components and layouts that make it easier to build modern, responsive Shiny apps, especially (but not just!) dashboards.

While writing the talk, I remembered an important change I made in my RStudio settings this year: adding library(bslib) to my shinyapp snippet.

RStudio snippets are little bits of text (e.g. shinyapp) that expand into larger blocks of code (e.g. a Shiny app skeleton) when you type them and press Tab or Shift + Tab. Here’s an example with a small snippet I frequently use: tdv expands into library(tidyverse).

RStudio snippets in action: I type tdv, press Tab and RStudio replaces tdv with library(tidyverse).

Before (default snippet)

For years, shinyapp has been the fastest way to start a quick little Shiny app. Here’s the default snippet that comes with RStudio:

snippet shinyapp
    library(shiny)

    ui <- fluidPage(
      ${0}
    )

    server <- function(input, output, session) {

    }

    shinyApp(ui, server)

It’s a great starting point! It also includes a neat feature of snippets: the ${0} placeholder marks the cursor location after the snippet expands. When you type shinyapp and press Tab, the app skeleton is added and the cursors is placed in the page function so you can start creating your UI right away.

After (with bslib)

Here’s my new shinyapp snippet with library(bslib) added.

snippet shinyapp
    library(shiny)
    library(bslib)

    ui <- page_${1:sidebar}(
      ${0}
    )

    server <- function(input, output, session) {

    }

    shinyApp(ui, server)

There are two changes:

I added library(bslib) to the top of the file.
I changed fluidPage() to page_sidebar() by default, but I used ${1:sidebar} to make it a placeholder in the template. After the text expands, the cursor moves to highlight sidebar so you can easily replace it with another page layout. Or you can press Tab to move to the ${0} position.

To add this snippet to RStudio, press Ctrl/Cmd + Shift + P to bring up the command palette. Then enter Edit code snippets and press Return. Scroll down to find the snippet shinyapp section and replace it with the code you can copy from the block above.

Bonus: A VSCode snippet

Snippets in VSCode work similarly to RStudio snippets. The only difference is that they are stored in JSON files, so the format of the snippet is slightly different.

To add your own shinyapp snippet to VSCode, press Ctrl/Cmd + Shift + P to open the VSCode command palette to run Snippets: Configure User Snippets. If you have an existing r.json snippets file, select that, otherwise select New Global Snippets file and enter r.json.

In the snippets file, add the following JSON. (If you already have R snippets, skip the outer { braces.)

{
    "shinyapp": {
        "prefix": "shinyapp",
        "body": [
            "library(shiny)",
            "library(bslib)",
            "",
            "ui <- page_${1:sidebar}(",
            "  ${0}",
            ")",
            "",
            "server <- function(input, output, session) {",
            "",
            "}",
            "",
            "shinyApp(ui, server)"
        ]
    }
}

Then, when you type shinyapp in a .R file and press Tab, VSCode will expand the snippet into the bslib version of the Shiny app skeleton.

Add last rendered or modified time to Quarto

Fri, 29 Mar 2024 04:00:00 GMT

A blog is a perpetual side-project machine

They say that if you copy-paste the same code three times, write a function. I’ve also heard that if you send the same email more than once, turn that email into a blog post.

But all the real work in life happens on Slack. So if a friend asks for your advice and you brain dump everything you know about the easiest way to install and manage Python, then you’ve got a first draft blog post on your hands!

They also say the hardest part of writing is staring at the blank page. But they are wrong. The hardest part of writing is figuring out how to start a new .qmd file in your blog repo without bumping into 3 to 10 other things you have been meaning to do.

Or worse, you might encounter that draft post you’ve been working on for months. Learn from my mistakes: it’s okay to stash that post, you can always tell yourself you’ll come back to it later!

You can probably tell by now that this is not a post about Python tooling for a happy setup. It’s also not a post about the few CSS tweaks I made while trying to open up this editor.

This post is sponsored by this a bit of YAML that caught my eye in my _quarto.yml file in the split second before I closed all my open tabs.

page-footer:
    left: >-
      © 2023 Garrick Aden-Buie

Now the fix could be easy. Consider this: I could change that file, commit it, and push it.

page-footer:
  left: >-
-    © 2023 Garrick Aden-Buie
+    © 2024 Garrick Aden-Buie

One GitHub Action, several minutes, and two Netlify notifications later my blog would be back to the future.

But this obviously won’t work. Time stops for no one. What will I do in March of 2025? Come back to this file and edit it by hand? Again? Like a caveman? Hard no.

Wouldn’t it be awesome if I could replace that 2023 that should be 2024 with something that turns into the current year whenever my blog rebuilds? I could save minutes in the entire lifetime of my blog! Imagine replacing the 2023 with something that would update itself automatically, like this:

page-footer:
  left: >-
    © {{< now year >}} Garrick Aden-Buie

It would be awesome, but that syntax doesn’t exist in Quarto. Or it didn’t until I my productive procrastination instincts kicked in!

Introducing now, a quarto extension for easily adding the time right now – err… the time right now when your Quarto blog or document is rendered.

Hello, now

To start using now in your own Quarto projects, just install the extension with this command.

quarto add gadenbuie/quarto-now

You can read all about the extension on its (quarto-built) page here: pkg.garrickadenbuie.com/quarto-now. Or keep reading this post for a quick intro.

Using now

now, the extension, comes with two shortcodes you can use just about anywhere: {{< now >}} and {{< modified >}}. Both work in the same way, except that {{< modified >}} will show the last modified time of the document, either by using the modified metadata field or the last modified timestamp of the file (on macOS and Linux only, sorry).

Here’s a quick example of the shortcodes in use. Note that in this post I added the modified date, which gives me a bit more control over what counts as a “modification”.

---
title: Add last rendered or modified time to Quarto
modified: 2024-03-29
---

This document was last modified at {{< modified >}}
and it was last rendered at {{< now >}}.

This document was last modified at 2024-03-29 00:00:00 and it was last rendered at 2026-03-08 16:12:24.

Formatting the time output

By default, now uses ISO 8601 (ish) formatting, but that’s not usually the most human-readable format. now comes with a few built-in aliases for common date and time parts.

Shortcode	Result	Format String
`{{< now >}}`	2026-03-08 16:12:24	`"%F %T"`
`{{< now year >}}`	2026	`"%Y"`
`{{< now month >}}`	March	`"%B"`
`{{< now day >}}`	08	`"%d"`
`{{< now weekday >}}`	Sunday	`"%A"`
`{{< now hour >}}`	04	`"%I"`
`{{< now minute >}}`	12	`"%M"`
`{{< now ampm >}}`	PM	`"%p"`
`{{< now date >}}`	03/08/26	`"%x"`
`{{< now time >}}`	16:12:24	`"%X"`
`{{< now datetime >}}`	Sun Mar 8 16:12:24 2026	`"%c"`
`{{< now isodate >}}`	2026-03-08	`"%F"`
`{{< now isotime >}}`	16:12:24	`"%T"`
`{{< now isodatetime >}}`	2026-03-08T16:12:24-0400	`"%FT%T%z"`
`{{< now timestamp >}}`	2026-03-08 16:12:24	`"%F %T"`

Note that the shortcodes only accept a single format argument, so you’d need to write {{< modified month >}} {{< modified year >}} instead of {{< modified month year >}}.

That said, you can always use the format strings supported by Lua’s os.date() function, so {{< modified "%B %Y" >}} gives you the month and year: March 2024. Note that because the format string has a space, it needs to be quoted "%B %Y".

You can find the complete list of accepted format strings in the now extension docs.

Have a good day, meow!

This was a fun excursion into the world of Quarto extensions! Thanks for reading and I hope you find now useful in your own projects. May you benefit from my inability to do the thing I’m “supposed to do” (hey, it’s my free time!).

The feature image was generated with DALL-E 3 and the following prompt: an m.c. escher style pencil sketch of a dreamy clock face.

Automated Image Overflow Detection for xaringan or remark Slides

Garrick Aden-Buie — Sun, 23 Jul 2023 04:00:00 GMT

Plain text slides

I’m a long-time fan of using the xaringan package to create slides¹ (as you might know from my blog). I’ve always appreciated that you can use R Markdown (or just regular markdown) to write your slides, but you can also use HTML, CSS and JavaScript to customize your slides. You can write in plain text, or you can turn each slide into a small web project.

Web-based slides can be shared with others online and tend to be more accessible than traditional slide formats. And with packages like renderthis – a package by John Helveston and me – you can even get PDF and other formats easily.

PDF slides and the case of the missing images

But PDF slides come with a few challenges of their own, most of which are related to the fact that you’re essentially using the browser’s printing features to print your slides to PDF.

The one that I’ve seen most often in PDFs is that images that happily render in the browser can suddenly disappear from the PDF version of the slides. Here’s how I described the problem in an issue in renderthis:

Chrome’s printing algorithm…excludes images that extend beyond the visible margin from printing. I’m sure [there’s a reason for this that] makes sense for general webpage printing, but it does cause problems for slides.

It can be hard to tell visually if the plot extends past the margins of the slide because the actual image may include empty space that you can’t see.

Often plots and other images include some empty space at the edges that can overflow the slide in a non-obvious way. The trick I tend to use is to add a red outline around all the images in the slides during development (or just before publishing the slides):

img, svg {
  outline: 1px solid red;
}

Example slides

I’ve created a small example slide deck that has a few slides with images that “accidentally” go beyond the edges of the slides. We’ll use this as our test case.

Source

---
title: Image overflow example
output:
  xaringan::moon_reader:
    seal: false
    self_contained: false
---

class: center middle

# First slide

```{r}
#| echo: false
xaringanExtra::use_share_again()
xaringanExtra::style_share_again(share_buttons = "none")
```

---

# Slide with clipped plot

This slide makes a plot that won't fit on the slide.

```{r plot, results="hide"}
hist(faithful$waiting)
```

---
class: bottom right

# Absolutely positioned plot



---

# SVG too big!



---

This little CSS can help during development,
not so much once you've built out all of your slides.

```{css eval = FALSE}
img, svg { outline: 2px solid red; }
```

```{css echo = FALSE}
/* this is the real css I applied
   to avoid red outlines in the share bar */
:not(.shareagain-button) > :is(img, svg) {
  outline: 2px solid red;
}
```

Automate it!

I absolutely love taking an annoying manual process and turning it into runnable, repeatable code. Automate all the things!

We can automate slide-image overflow with these 4 steps:

Fire up headless Chrome and load the slides in the virtual, remote-controllable browser.
Determine the natural desired size of the slides (in pixels) and then set the virtual browser window to this size. This isn’t strictly necessary, but it makes the math much easier later on, so it’s worth it.
Make all of slides visible at once. Otherwise, remark (the JavaScript library that powers the slides) hides all of the slides except the current slide. Elements that are hidden from the browser don’t have a size, ruining our ability to detect overflow.
Finally, find all of the images in the slides and check if any escape the slide (browser window) boundaries.

With chromote

Now we get to the fun part. First we need to start up chromote and initialize a new browser session.

library(chromote)

chrome <- ChromoteSession$new()

I’ve stored the example slide deck in a folder called example-slide, and I’ve already rendered it from index.Rmd to index.html. To load the file in Chrome, we need the full path to the file, prefixed with file://, e.g. file:///Users/garrick/.../example-slide/index.html.

slide_path <- file.path("example-slide", "index.html")
slide_path_uri <- file.path("file:/", normalizePath(slide_path))

Now we can load the slides in Chrome. I’m using a small trick here to make sure that the R session waits for the browser to load the page. The trick is to request the Page.loadEventFired event via chromote asynchronously (wait_ = FALSE), then load the page, after which the we block the R session until the page load event fires.

pl <- chrome$Page$loadEventFired(wait_ = FALSE)
chrome$Page$navigate(slide_path_uri, wait_ = FALSE)
#> 

# Block until the page load event fires
chrome$wait_for(pl)
#> $timestamp
#> [1] 195025.8

That’s not something you usually have to do with chromote when you’re interacting with pages on the web (because those take long enough to load), but local pages load very quickly, sometimes faster than the R expressions are evaluated. (In other words, if you execute the navigate command and then wait for page load event, you might miss it.)

Match window size to slide size

Now that we have our slides loaded in the headless browser, we need to figure out what size they are. remark uses some rather random pixels values based on the specified slide ratio, so the easiest thing to do is find the first visible slide and figure out what size remark used.

Here’s a small JavaScript function that will do that for us:

function get_slide_size() {
  // Find the first visible slide in the deck
  const visible_slide = document
    .querySelector('.remark-visible .remark-slide-scaler')

  // and inspect it's width and height
  // knowing that remark will set these values
  const {width, height} = visible_slide.style

  return {
    width: parseInt(width),
    height: parseInt(height)
  }
}

We the use the $Runtime$evaluate() methods of the chromote object to run the JavaScript function definition, by passing the JS code as a string².

Loading the function in the browser via chromote

Read and Visualize your Twitter Archive

Garrick Aden-Buie — Wed, 07 Dec 2022 05:00:00 GMT

Twitter finds itself in an… interesting… transition period. Whether or not you’re considering jumping ship to another service — you can find me lurking on Mastdon — you should download an archive of your Twitter data. Not only does the archive include all of your tweets, it also contains a variety of other interesting data about your account: who you followed and who followed you; the tweets you liked; the ads you were served; and much more.

This post, very much inspired by the awesome Observable notebook, Planning to leave Twitter?, shows you how to use R to read and explore your archive, using my own archive as an example.

Read on to learn how to read your Twitter archive into R, or how to tidy your tweets. The second half of the post showcases a collection of plots about monthy tweet volume, popular tweets, the time of day when tweets were sent, and the app used to send the tweet.

I’ve also included a section on using rtweet to collect a full dataset about the tweets you’ve liked and another section about the advertising data in your Twitter archive.

Reading your Twitter archive

Get your Twitter data archive

First things first, you need to have your Twitter data archive. If you don’t have it yet, go to Settings and Privacy and click Download an archive of your data. After you submit the request, it takes about a day or so for an email to show up in your inbox.

@grrrck your Twitter data is ready

Your Twitter archive is ready for you to download and view using your desktop browser. Make sure you download it before Nov 12, 2022, 9:46:31 PM

The archive downloads as a zip file containing a standalone web page — called Your archive.html — for exploring your data. But the real archive lives in the included data/ folder as a bunch of .js files. I’ve copied that data/ directory into my working directory for this post.

Setup

On the R side, we’ll need the usual suspects: tidyverse and glue.

library(tidyverse)

#> ── Attaching core tidyverse packages ───────────────────────
#> ✔ dplyr     1.0.10     ✔ readr     2.1.3 
#> ✔ forcats   0.5.2      ✔ stringr   1.5.0 
#> ✔ ggplot2   3.4.0      ✔ tibble    3.1.8 
#> ✔ lubridate 1.9.0      ✔ tidyr     1.2.1 
#> ✔ purrr     1.0.1      
#> ── Conflicts ────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()
#> ℹ Use the conflicted package () to force all conflicts to become errors

library(glue)

(I’m using the dev version of tidyverse (1.3.2.9000), which loads lubridate automatically, and the dev version of purrr that is slated to become version 1.0.0.)

To read in the data files, I’ll use jsonlite to read the archive JSON data, with a small assist from brio for fast file reading. I’m also going to have some fun with ggiraph for turning static ggplot2 plots into interactive plots.

Finally, the Twitter archive doesn’t require API access to Twitter, but you can use it to augment the data in the archive. The rtweet package is excellent for this, even though it takes a little effort to get it set up.

Read the manifest

The data/ folder is surprisingly well structured! There are two key files to help you find your way around the archive. First, the README.txt file explains the structure and layout of the files, and includes descriptions of the data contained in all of the files.

Here’s how the README describes the account.js data file:

account.js
- email: Email address currently associated with the account if an email address has been provided.
- createdVia: Client application used when the account was created. For example: “web” if the  account was created from a browser.
- username: The account’s current @username. Note that the @username may change but the account ID will remain the same for the lifetime of the account.
- accountId: Unique identifier for the account.
- createdAt: Date and time when the account was created.
- accountDisplayName: The account’s name as displayed on the profile.

The data/ folder also contains a manifest.js file that can be used to help read the data included in the archive. Let’s start by assuming this file is JSON and reading it in.

jsonlite::fromJSON("data/manifest.js")
#> Error in parse_con(txt, bigint_as_char): lexical error: invalid char in json text.
#>                                        window.__THAR_CONFIG = {   "use
#>                      (right here) ------^

Here we hit our first snag. The archive files are packaged as JSON, but they’re not strictly compliant JSON files; they include some JavaScript to assign JSON objects to the global namespace (called window in the browser). Here’s the data/manifest.js file as an example.

window.__THAR_CONFIG = {
  // ... data ...
}

If we just remove everything up to the first the { or sometimes [ on the first line, we can turn the data into valid JSON.

lines[1] <- sub("^[^{[]+([{[])", "\\1", lines[1])
manifest <- jsonlite::fromJSON(lines)

This worked, but… jsonlite was designed for statistical work, so it transforms the data structure when reading in the JSON. For example, by default it converts arrays that look like JSON-ified data frames into actual data.frames.

manifest$dataTypes[1:2] |> str()
#> List of 2
#>  $ account          :List of 1
#>   ..$ files:'data.frame':    1 obs. of  3 variables:
#>   .. ..$ fileName  : chr "data/account.js"
#>   .. ..$ globalName: chr "YTD.account.part0"
#>   .. ..$ count     : chr "1"
#>  $ accountCreationIp:List of 1
#>   ..$ files:'data.frame':    1 obs. of  3 variables:
#>   .. ..$ fileName  : chr "data/account-creation-ip.js"
#>   .. ..$ globalName: chr "YTD.account_creation_ip.part0"
#>   .. ..$ count     : chr "1"

That’s often quite helpful! But I find it’s safer, when trying to generalize data reading, to disable the simplification and know for certain that the data strcutre matches the original JSON. For that reason, I tend to disable the matrix and data.frame simplifications and only allow jsonlite to simplify vectors.

Here’s a quick helper function that includes those setting changes and the first line substitution needed to read the archive JSON files.

read_archive_json <- function(path) {
  lines <- brio::read_lines(path)
  lines[1] <- sub("^[^{[]+([{[])", "\\1", lines[1])

  jsonlite::fromJSON(
    txt = lines,
    simplifyVector = TRUE,
    simplifyDataFrame = FALSE,
    simplifyMatrix = FALSE
  )
}

Now we’re ready to read the manifest again.

manifest <- read_archive_json("data/manifest.js")
names(manifest)
#> [1] "userInfo"    "archiveInfo" "readmeInfo"  "dataTypes"

The manifest file contains some information about the user and the archive,

str(manifest$userInfo)
#> List of 3
#>  $ accountId  : chr "47332433"
#>  $ userName   : chr "grrrck"
#>  $ displayName: chr "garrick aden-buie"

plus details about all of the various data included in the archive, like the data about my account.

str(manifest$dataTypes$account)
#> List of 1
#>  $ files:List of 1
#>   ..$ :List of 3
#>   .. ..$ fileName  : chr "data/account.js"
#>   .. ..$ globalName: chr "YTD.account.part0"
#>   .. ..$ count     : chr "1"

Each dataType in the manifest points us to a file (or files) in the archive and helpfully tells us how many records are included.

Here are the data files with the most records.

Code: Manifest, Top Records

countdown v0.4.0 – Now on CRAN!

Garrick Aden-Buie — Mon, 15 Aug 2022 04:00:00 GMT

I’m stoked to announce that countdown is now available on CRAN! Countdown to something awesome in xaringan, Quarto, R Markdown, or Shiny.

In this post, I reflect a bit on the development of countdown, but you can also skip straight to the release notes!

What is countdown?

01:30

countdown() is a simple timer you can use in presentations, documents and Shiny apps. It’s great for teaching or breakout sessions!

👈☝️ Click the timer to start it. Click again to pause. Double click to reset it. Adjust the timer on the fly with the + and − buttons.

Everything you need to know about countdown, you can learn from the docs-slash-presentation at pkg.garrickadenbuie.com/countdown.

01:30

countdown::countdown(
  minutes = 1,
  seconds = 30,
  warn_when = 30
)

Installing countdown

Installing countdown is now a whole lot easier:

install.packages("countdown")

As always, you can still get the latest and greatest in-development versions from GitHub

# install.packages("remotes")
remotes::install_github("gadenbuie/countdown")

or from gadenbuie.r-universe.dev.

options(repos = c(
  gadenbuie = 'https://gadenbuie.r-universe.dev',
  CRAN = 'https://cloud.r-project.org'
))

install.packages('countdown')

A brief history of countdown

Before we talk about all the new things in countdown, I want to take a small minute to get nostalgic. I hope you don’t mind indulging me (or skip ahead if you’d rather get right to business).

rstudio::conf(2019)

In 2019 I went to rstudio::conf in Austin, TX where a highlight of the conference, for me, was the Train-the-Trainer: Tidyverse Track workshop by Garrett Grolemund and Greg Wilson. That workshop specifically marked a turning point in my career and I left rstudio::conf very inspired to build and teach cool things in R.

I also walked away from rstudio::conf(2019) with another key take away: it was time to learn JavaScript. An odd thing to take away from an R conference, yes. (Although I don’t think I’m alone in this kind of realization; this year many people left rstudio::conf(2022) thinking that it’s time to learn Python.)

These two inspirations came together in my first post-conf project: a countdown timer for xaringan slides.

A slide from Garrett’s workshop materials with a 4-minute timer in the lower right corner.

Garrett used timers extensively to pace break out sessions and they worked surprisingly well to keep everyone on track. One funny thing I noticed during our workshop session was that Garrett would frequently have to switch to slide-edit mode (in Keynote, I think) to fiddle with the timer as he adjusted the length of the “your turn” session. This is pretty normal; an instructor probably has a sense of approximately how long an activity will take and we’ll often will adjust the time spent on the activity based on how the audience is doing, how well the material is working, or how close to lunch or a break we are in the session.

So my idea was to build a countdown timer that you could drop into a slide and easily use to time an event. I also wanted to make it easy to adjust the time, but my JavaScript skills were limited to what I could learn from StackOverflow, so I compromised and decided that you could only bump the timer up. After all it’s not like you have to end the timer, you can always just move on in your slides.

It becomes an R package

I cobbled together an R package that was a fairly decent R interface around a collection of lines of JavaScript that I barely understood, that somehow assembled into an actual working timer. I made a cool intro-slash-docs presentation and would probably have sat on it for a while longer if it weren’t for Mara Averick who spotted my GitHub activity and soft-announced the package for me.

Not long after that, and slightly to my horror (please don’t go looking at my JavaScript code), Hadley submitted an issue. Actually, two issues. Obviously, that was an exciting turn of events. His suggestions were solid and helped improve the quality of the timer: he suggested a warning state and a full-screen view/app.

Amazingly, the package worked! People really seemed to like it, it solved a niche but useful need that many people have when teaching, and it let me learn a ton about how to build htmlwidgets in R. I’m proud of the R interface — it’s easy to use and configure — and I think the feature set hits the right balance of looking good right out of the box without doing too much.

But that JavaScript code…

Since I wrote the first version of countdown, I’ve learned a whole lot more JavaScript and I know a whole lot more about how to build web things in R. countdown’s underlying code has always haunted me a little, but on the other hand it was chugging away, still working fine for most people in most situations.

So I left it alone…

Screenshot of the countdown GitHub repository page where the phrase “3 years ago” is highlighted. That is GitHub’s summary of the last time I updated countdown.

…for almost 3 years.

And wow how much has changed in the three plus years since rstudio::conf(2019). Not only did I lead a workshop about JavaScript for Shiny users at rstudio::conf(2020) (hashtag js4shiny), and not only do I now work for RStudio¹, but I was also part of the program committee for conference planning. Which means I saw colleagues were still using my countdown timer in workshop slides.

And that old franken-JavaScript code still haunted me.

So this year, in part inspired by the return of the final rstudio::conf, I decided that finally rewriting that JavaScript would be the perfect conference side-hack project.

Which led to countdown v0.4.0 arriving on CRAN!

coundown v0.4.0

The Old JavaScript

My first implementation relied heavily on the JavaScript function setTimeout, which takes a function and a delay in milliseconds: setTimeout(function, delay). When called, the browser waits until the delay is over and then calls the function.

A neat trick with setTimeout is that you can call it recursively inside a function as a way to run that function repeatedly. Below I’ve defined a function timerTick() that moves the timer forward one tick. It also uses setTimeout to schedule the next tick of the timer. And by using a delay of 1000 milliseconds, I’ve set up a function that runs once per second — just like a clock 😉.

This is, essentially, how countdown worked before. For each run of timerTick(), I would decrement the number of remaining seconds by one and update the timer. If there’s time left on the timer, then timerTick() shcedules another update for 1 second later. If there isn’t any time left, we can stop the timer by simply not scheduling any more timer updates.

function timerTick() {
  const timer = document.getElementById('timer')

  // update the timer
  timer.value -= 1
  console.log(`${timer.value}s remaining...`)

  if (timer.value > 0) {
    // there's time left, schedule next tick
    setTimeout(timerTick, 1000)
  } else {
    // time is up, reset the timer
    console.log(`Time's up!`)
    timer.classList.remove('running')
    timer.innerText = 'Start Timer'
    // notice we don't schedule another tick
  }
}

And this works^*! Try it out by clicking the button below.

^*Almost. This almost works. It works pretty well if you start the timer and then don’t touch the browser window or switch to another tab. So it does usually work fine when you’re presenting slides.

But it turns out that setTimeout() is more like suggestThatThisRunsLater(). There’s really no guarantee that the function you scheduled to run 1,000 milliseconds from now is actually going to run in 1,000 milliseconds.

There are many things that can get in the way of that function being run when you expect it. If you move to a different tab and come back, for example, there’s no guarantee that the background tab would keep chugging along, running my function every seconds. Browsers have better things to do and they’ll de-prioritize pages that aren’t being actively shown to users. This means that sometimes setTimeout(fn, 1000) runs fn 1 second from now, but depending on what else the browser is doing it could be a lot longer than that.

So how do we get around this? 🤔

All New JavaScript

The new JavaScript version of countdown does something really simple. It doesn’t rely on setTimeout() directly to keep track of the time.

Yes, it still schedules the next tick on 1 second intervals, but it doesn’t trust that exactly one second has passed. Now, when the user starts the timer, countdown will note when the timer should end and recalculates the remaining time with each tick. This means the updates are always accurate, even if there happen to be 4 seconds between consecutive ticks.

It also means that we can bump a running timer up or down by moving that end time later or earlier. To stop the timer, we just note how much time is left and to restart it again we recalculate the end time based on how much time was left when we paused.

There’s a small amount of internal state to keep track of, which happens to basically cry out for a JavaScript class. So the new countdown timer is implemented via a CountdownTimer class.

Here’s a sketch of the class containing three core methods:

tick() runs the timer, like above, except this time we calculate the remaining number of seconds on each tick. When there are less than 0 seconds left, we call the finish() method.
start() gets things started by calculating when the timer should end and kicking off the tick() method.
finish() wraps up by resetting the timer.

class Timer {
  constructor(el, duration) {
    // The timer's attached to a button
    this.element = el
    // it has a duration that's set when initiated
    this.duration = duration
    // and it will have an end when running
    this.end = null
  }

  tick () {
    // decide and report how much time is left
    const remaining = this.end - Date.now()
    console.log(`${remaining / 1000}s remaining...`)

    // and then schedule the next tick or finish up
    if (remaining > 0) {
      setTimeout(this.tick.bind(this), 1000)
    } else {
      this.finish()
    }
  }

  start () {
    if (this.end) return

    console.clear()
    this.element.innerText = 'Timer is running...'
    // the timer ends duration (s) * 1000 (ms) from now
    this.end = Date.now() + this.duration * 1000
    this.tick()
  }

  finish () {
    this.end = null
    this.element.innerText = 'Start Timer'
    console.log(`Time's up!`)
  }
}

Run the 5-second timer by clicking the button above. Notice that even though we used the same setTimeout(code, 1000) as before to schedule each tick for one second later, because this version precisely reports how much time is left you can see that our timer drifts a bit away from running perfectly once per second.

New buttons and keyboard interactions

Beyond the improved timer, the new CountdownTimer class makes it a whole lot easier to add additional features that need to build on the timer’s internal state.

For example, you can now

Click to start or stop the timer
Double click to reset
Bump the timer up or down using the + and − buttons
Do all of the above with keyboard shortcus:
- Space or Enter to start or stop the timer
- Esc to reset
- ↑ or ↓ to bump up or down

Shiny!

The shiny new countdown package also has plenty of Shiny features. Countdown timers can be controlled directly from Shiny with countdown_action() or countdown_update() and timers are now also inputs that report their state!

You can find an example Shiny app with a timer, plus an explanation of how it all works, by running

to launch an example app. The example app is also available on my website at apps.garrickadenbuie.com/countdown-shiny-example.

In a nutshell, the timer will report its state using its input id. For example, countdown(id = "timer") will report its state to Shiny via input$timer. The input reports the event that caused the state to change and the state of the timer:

Here’s another small app that demonstrate how you could use a button to toggle the state of the timer.

Improved countdown App

A screenshot of the full screen countdown timer app.

All of the Shiny updates mentioned above are used to power countdown_app(), a full screen Shiny app for running timers. These work really well for timing speakers at conferences or for a quick way to keep track of a break out session in workshops or meetings.

The app itself received a few upgrades, most importantly is the ability to share a timer with the settings you want using the URL. This uses Shiny’s Bookmarking state features to save your settings in the URL and restore them when you load that link.

For example, this timer is a 20 minute timer with a warning at 5 minutes that updates every 10 seconds.

New Options

Finally, countdown gained a new option. You can now start the timer as soon as it is visible by setting start_immediately = TRUE. The “as as soon as it’s visible” works pretty well: in xaringan and Quarto slides it starts when you land on the slide and in regular HTML documents the timer starts when you scroll the timer into view.

It’s also worth mentioning that countdown now uses prismatic for color calculations. I was really happy to see that Emil added best_contrast() and switching to use that function cleaned up a lot of internal code for me!

Thank you!

Huge thanks to the many people who opened issues or contributed code to countdown over these years. You all rock 🧡

@andrewpbray, @apreshill, @ConnorJPSmith, @csgillespie, @Dr-Joe-Roberts, @fvitalini, @hadley, @HaoZeke, @jhelvy, @jvcasillas, @moshpirit, @rtheodoro, @sje30, @spcanelon,and @thiyangt.

If you’ve read this far, thank you! Thanks for using countdown and making developing R packages fun. Reach out in the comments or on Twitter (I’m @grrrck) with any questions or thoughts ☺️

Footnotes

At least until October: RStudio is becoming Posit.↩︎

Process Profile Pictures with magick

Garrick Aden-Buie — Tue, 12 Jul 2022 04:00:00 GMT

rstudio::conf(2022) runs on R

rstudio::conf(2022) is just around the corner! This year, I’ve been fortunate to be part of the conference program committee, the small group of RStudio people who gather and evaluate talk submissions, make the tough scheduling decisions about the sessions and talks in the conference program, and generally wrangle and herd all the speaker and talk information into the final schedule.

My favorite part of the process — apart from seeing all of the wonderful and creative ways our community approaches data science with R (okay, and Python too) — is finding out how many creative ways we use R to manage the conference. Let’s just say Jenny Bryan is a literal Google Forms/Sheets/Drive wizard.

One of the things I love about R is the cycle of starting a task wondering Can I do this with R? and ending with Wow, I can do this with R! I’ve been using R for a while and I’m still occasionally surprised when I find myself on this virtuous loop.

This post is about an otherwise mundane conference admin task that would have involved a lot of manual labor (in the form of clicks and mouse movements) that I automated with the help of a few R packages. Maybe in future posts I’ll share more cool things that we did with R in the making of rstudio::conf.

Oh and I hope to see you there, either in person, online or on Twitter at #RStudioConf2022! Learn more at rstd.io/conf.

Send me your profile picture, please

Suppose you’ve asked 100-ish people to send you a profile picture and to your surprise they all followed through and sent you an actual image.

But, of course, you now have a new problem. Each of those 100-ish people has used slightly different sizes for their profile picture. They’re all sorts of different shapes, sizes, and resolutions.

Some people’s profile images feature their faces, centered and tightly cropped. Others are photographed at a distance or off-center.

An example profile picture you received. Image by christian buehner.

In their final placement, you want all of the profile images to be circular images centered on the person’s face as much as possible. If we took the image above and simply centered it inside a circle, we would get something like this:

Obviously, we’d rather not clip half of the person out of their profile image, so we’ll need to edit this photo. But there are hundreds of them and most of them will need some adjustment.

Good news! You have access to R, where we can use tools like magick to read and process the images, or face detection with neural networks. So with a few hours of work you can save yourself an hour of mindless clicking. Let’s do this!

Collect the Profile Pictures

To see how this all works, I’ve downloaded four example profile pictures from unsplash¹.

dir.create("profiles")
photo_ids <- c("DItYlc26zVI", "bpxgyD4YYt4", "6anudmpILw4", "3dqSZidOkvs")

for (id in photo_ids) {
  download.file(
    sprintf("https://source.unsplash.com/%s", id),
    sprintf("profiles/%s.jpg", id)
  )
}

I’ve put the photos in a profiles/ directory so that I can list the them all at once:

fs::dir_ls("profiles")
## profiles/3dqSZidOkvs.jpg profiles/6anudmpILw4.jpg profiles/DItYlc26zVI.jpg 
## profiles/bpxgyD4YYt4.jpg

Magick with R

The first step is to use the magick package to read in our profile pictures.

library(magick)
library(purrr)

profiles <-
  fs::dir_ls("profiles") |>
  map(image_read)

profiles
## $`profiles/3dqSZidOkvs.jpg`
##   format width height colorspace matte filesize density
## 1   JPEG  1080    810       sRGB FALSE   135368   72x72
## 
## $`profiles/6anudmpILw4.jpg`
##   format width height colorspace matte filesize density
## 1   JPEG  1080    720       sRGB FALSE    46181   72x72
## 
## $`profiles/DItYlc26zVI.jpg`
##   format width height colorspace matte filesize density
## 1   JPEG  1080    720       sRGB FALSE    87827   72x72
## 
## $`profiles/bpxgyD4YYt4.jpg`
##   format width height colorspace matte filesize density
## 1   JPEG  1080    608       sRGB FALSE    80277   72x72

Here are the four profiles. As you can see, they come in a variety of sizes and the person in the frame is rarely centered.

Finding Faces

Now for the most fun of this entire post. After a quick search on r-pkg.org, I found a few packages that provide methods for facial detection; I tried image.libfacedetection first and it worked out so well that I didn’t have to look any further on the list.

As it says on the CRAN page, image.libfacedetection is

An open source library for face detection in images. Provides a pretrained convolutional neural network based on https://github.com/ShiqiYu/libfacedetection which can be used to detect faces which have size greater than 10x10 pixels.

The best feature — apart from reliably detecting faces — is that it works really well with magick. The core functionality is all wrapped up in a single function, image.libfacedetection::image_detect_faces(), and the example in the README tells you just about everything you need to know.

In short, after reading the image into R with magick::image_read(), you can call image_detect_faces() to find faces in the image. image_detect_faces() returns data about the detected faces, and you can use its plot() method to overlay boxes over the found faces in the image.

library(image.libfacedetection)
faces <- all_profiles |> image_detect_faces()
plot(faces, all_profiles, only_box = TRUE)

image_detect_faces() returns some interesting data about the detected faces:

The data frame detections indicates the locations of these. This data.frame has columns x, y, width and height as well as a column called confidence. The values of x and y are the top left of the start of the box.

faces
## $nr
## [1] 4
## 
## $detections
##     x   y width height confidence landmark1_x landmark1_y landmark2_x
## 1 153  48    43     57         99         158          70         175
## 2 140 239    43     58         99         150         260         171
## 3 365  64    33     44         99         373          79         387
##   landmark2_y landmark3_x landmark3_y landmark4_x landmark4_y landmark5_x
## 1          68         162          80         162          92         177
## 2         260         160         272         151         282         168
## 3          80         377          86         372          94         385
##   landmark5_y
## 1          89
## 2         282
## 3          95
##  [ reached 'max' / getOption("max.print") -- omitted 1 rows ]
## 
## attr(,"class")
## [1] "libfacedetection"

Since we asked for a profile picture, we can reasonably expect that there’s only one person in the image. So we’ll take the detection with the highest confidence (in case something else registers as a face), and find the center of the detected region.

find_face_center <- function(image) {
  detections <- image.libfacedetection::image_detect_faces(image)$detections
  best_face <- which(detections$confidence == max(detections$confidence))
  dims <- as.list(detections[best_face[[1]], ])
  list(
    x = dims$x + dims$width / 2,
    y = dims$y + dims$height / 2
  )
}

So when applied to our example profile image, we find that our subject’s face is centered at (697.5, 290).

face_center <- find_face_center(profiles[[3]])
str(face_center)
## List of 2
##  $ x: num 698
##  $ y: num 290

In the next steps, we’ll resize and crop the photo so that it’s centered, as much as possible, on this point.

Resize

Our goal is to resize and crop the photo into an 600px square image. If we start with an image smaller than 600px in either dimension, then we won’t scale up. We also take another shortcut: since most people will provide a profile image that prominently features their face, we can start by shrinking the smaller side of the image down to match the desired image size.

This shortcut keeps us from perfectly framing the person’s face. Sometimes their face is too close to the edge of the picture, and in other cases there may be negative space around their head that will end up in the cropped profile image. I’d argue that this is okay. It keeps our cropping from being too perfect and the final images still retain some of the character of the original photo.

Our example profile image is 1080px wide and 720px tall, so we’ll resize the image proportionally down to an image with height 600px.

resize_fit <- function(image, size = 600) {
  info <- image_info(image)
  size <- min(size, info$width, info$height)
  image_resize(
    image,
    geometry_size_pixels(
      height = if (info$width >= info$height) size,
      width = if (info$height > info$width) size
    )
  )
}

When applied to our example profile image, we end up with a 900px × 600px image.

resized_profile <-
  profiles[[3]] |>
  resize_fit()

resized_profile |> image_info()
##   format width height colorspace matte filesize density
## 1   JPEG   900    600       sRGB FALSE        0   72x72

In the next step, we’ll figure out which 600px horizontal region best covers the person’s face.

Find Resized Faces

Wait. I showed the face-center discovery step above because it’s the coolest part of this pipeline, but we don’t actually perform the facial detection first. We need to know where the person’s face is located after we scale down their profile image.

resized_profile |>
  find_face_center()
## $x
## [1] 579
## 
## $y
## [1] 240.5

Cropping

Now that we know where the center point of the person’s face is located in the image, and also because we’ve already resized the image so we don’t have to worry about its height, we only need to crop the image in one direction. The problem now is that we need to pick a 600px width region within the full 900px range.

           point
|------[=====*=====]---|
       ^~~ width ~~^
^......................^ range

This isn’t too complicated. There are three cases:

The point is so close to the start of the range that it we can’t center the point in our width and instead have to start at 0.
Similarly, the point might be so close to the end of the range that our crop width lines up with the end. Or, in other words, the crop width starts at range - width.
Or finally, we can center the point in our crop width, so it should start at point - width/2.
Oh, and there’s an edge case: if the width is greater than or equal to the full range, then the offset is 0, too.

This logic gives us the following crop_offset() function:

crop_offset <- function(point, range, width) {
  # 4. Catch the edge case first
  if (width >= range) return(0)

  if ((point - width / 2) < 0) {
    # 1. must start at left edge
    return(0)
  }
  if ((point + width / 2) > range) {
    # 2. must start at right edge
    return(range - width)
  }
  # 3. enough space on both sides to center width in range
  point - width / 2
}

Which in our example case tells us that we could crop our resized profile image to a 600px square, offset by the following amount in the x direction:

offset <- crop_offset(
  point = 579,
  range = 900,
  width = 600
)
offset
## [1] 279

We can use magick::image_crop() with the magick::geometry_area() helper function:

When this image is used as a profile or avatar picture, it ends up looking much better than the uncropped and uncentered version!

Put it all together

The last step is to put everything we’ve sketched out above into a single function that takes a magick image and returns a new cropped and centered version. And here’s that function.

resize_crop_to_face <- function(image, size = 600) {
  image <- resize_fit(image, size)
  info <- image_info(image)

  # size may have changed after refit
  size <- min(info$height, info$width)

  is_image_square <- info$width == info$height
  if (is_image_square) {
    return(image)
  }

  face <- find_face_center(image)

  image_crop(
    image,
    geometry = geometry_area(
      width = size,
      height = size,
      x_off = crop_offset(face$x, info$width, size),
      y_off = crop_offset(face$y, info$height, size)
    )
  )
}

Starting over from the beginning, we can read all of the profile images and resize and crop them around the subject’s face in just a few lines

profiles <-
  fs::dir_ls("profiles") |>
  map(image_read) |>
  map(resize_crop_to_face)

and then we can write them back into the profiles directory.

fs::dir_create("profiles_cropped")

profiles |>
  iwalk(function(image, path) {
    new_path <- fs::path("profiles_cropped", fs::path_file(path))
    image_write(image, new_path)
  })

The end result is four perfect profile pictures!

Footnotes

Images by christian buehner, Eunice Lituañas, Foto Sushi, and Eye for Ebony.↩︎

xaringanExtra v0.6.0 — Now on CRAN!

Garrick Aden-Buie — Tue, 07 Jun 2022 04:00:00 GMT

I’m delighted to announce that xaringanExtra is now available on CRAN! Bring something extra to your xaringan slides.

This is the first release available via install.packages() and it includes a new extra, plus some bug fixes. Read on to learn more!

What is xaringanExtra?

What can xaringanExtra do for you and your xaringan slides? A whole lot! Here are all the things you can do with xaringanExtra:

Add an overview of your presentation with tile view
Make your slides editable
Share your slides in style with share again
Broadcast your slides in real time to viewers with broadcast
Scribble on your slides during your presentation with scribble
Announce slide changes with a subtle tone
Animate slide transitions with animate.css
Add tabbed panels to slides with panelset
Add a logo to all of your slides with logo
Add a top or bottom banner to all of your slides with banner
Add a search box to search through your slides with search
Use the Tachyons CSS utility toolkit
Add a live video feed of your webcam
Add one-click code copying with clipboard
Always play gifs from the start with freezeframe
Fit your slides to fill the browser window
Add extra CSS styles

But wait, there’s more! These features aren’t limited to just xaringan slides. A bunch of the features can be used in regular R Markdown documents, in particular: panelset, animate.css, Tachyons, and clipboard.

Installing xaringanExtra

Installing xaringanExtra is now a whole lot easier:

install.packages("xaringanExtra")

As always, you can still get the latest and greatest in-development versions from GitHub

# install.packages("remotes")
remotes::install_github("gadenbuie/xaringanExtra")

or from gadenbuie.r-universe.dev.

options(repos = c(
  gadenbuie = 'https://gadenbuie.r-universe.dev',
  CRAN = 'https://cloud.r-project.org'
))

install.packages('xaringanExtra')

Wait, it’s on CRAN now?

Yeah, can you believe it?!

When I started working on xaringanExtra (checks notes) 943 days ago — that’s 2 ½ years or 13 pandemic years ago — I never expected that xaringanExtra would ever be anything beyond a toy package I used for fun ~~procrastination~~ experiments while preparing slides for talks and workshops.

I’m honored and grateful that so many other people have found the package helpful and have used it to make their presentations more engaging! I’m also humbled that when the Quarto team developed the next generation of slide-building tools, they made a point to ensure that Quarto presentations include, out-of-the-box, many of the features from xaringanExtra.

I’d also like to thank the community of users and developers who have reported issues or shared xaringanExtra on social media or talked about it in workshops and talks. A big up-front thank you to Matt Warkentin for developing one of my favorite extras: 🧑‍🎨 Scribble!

Sincerely, thank you to the 54 awesome people who contributed issues, advice, fixes or features since I started working on xaringanExtra:

@albert-ying, @apreshill, @BerriJ, @brianmsm, @chainsawriot, @ColinConwell, @dataning, @dmi3kno, @drfurtado, @dunstone-a, @evanmorier, @giabaio, @GitHunter0, @gpapageorgiou, @gtalckmin, @howardm, @ignacio82, @issactoast, @jhelvy, @jmgirard, @jooyoungseo, @juandodyk, @jvcasillas, @kim-soo-hwan, @koliajaykr, @konstruktur, @LauraRK, @mattwarkentin, @MayaGans, @mchiapello, @mfherman, @mine-cetinkaya-rundel, @nithinmkp, @nucleic-acid, @pat-s, @pomang-211, @psads-git, @py9mrg, @realauggieheschmeyer, @RRMaximiliano, @rsimonmd, @ryanstraight, @SchmidtPaul, @simonedu, @skamper1, @spcanelon, @Sumidu, @tgerke, @TuQmano, @uriahf, @W-Mohammed, @yonicd, @youngwoos, and @yyzeng.

A new extra: banner!

One new feature made it into this release: 📰 banner!

The idea behind banner is simple. Sometimes you want to have some text appear on every slide in your presentation (or at least most of the slides).

You might want to show a short link to the slides being presented. Or you might want to add your name or talk title in the slide margin so that attendees who missed your intro or don’t have the conference schedule in front of them know who you are and what you’re talking about.

xaringanExtra::use_banner() solves this problem. Just add a chunk like the one below to your slides

```{r xaringan-banner, echo=FALSE}
xaringanExtra::use_banner(
  top_left = "My Awesome Talk Title",
  top_right = "Mr. Fancy Pants",
  bottom_left = "bit.ly/my-awesome-talk",
  exclude = "title-slide"
)
```

and you’ll get slides that look kind of like this!

Saving Daylight Time?

Garrick Aden-Buie — Sat, 12 Mar 2022 05:00:00 GMT

Remember when Daylight Saving Time happened to us again? You know, that day that causes us all to grumble loudly about the ridiculousness of our biannual clock adjustment and loss of sleep?

In this post, I engage in some self-care data visualization to explore day light hours in cities across the world and the United States, inspired by an awesome night hours series by Krisztina Szucs.

Are we saving daylight in Atlanta, GA?

As a parent, I loathe daylight saving time. Nothing reveals the shared delusion of time like trying to explain to your children why we moved the clock forward an hour when it means they’ll suddenly need to go to bed while it’s light out or wake up and get ready for school while it’s dark.

So in this spirit, I started to wonder: what kind of returns are we getting on our daylight saving? Except, rather than try to directly answer that question — since that’s way too hard — I chose to visualize day light hours to see how they align with the modern work day.

The plot below shows the yearly day light schedule for 2022 in Atlanta, GA where I live. We’ll also take a look at the day light schedule in other cities around the world or across the United States. At the end of this post, I’ll share the code I used to make the plot below.

Atlanta, GA.

	Date	Sunrise	Sunset	Daylight	Non-Work
Shortest Day	Dec 21	7:41 am	5:38 pm	9h 56m	1h 56m
Longest Day	Jun 21	6:32 am	8:54 pm	14h 21m	6h 21m

It’s pretty clear from this visualization that in Atlanta, GA, which is very much on the western edge of the U.S. Eastern time zone, year-round standard time is a decent way to live life. What about other cities in the world?

Around the World

Pick a city

A sunrise/sunset time plot.

Across the US

It seems that every year we talk about finally doing something about daylight saving time, but this year the U.S. Senate actually went so far as to pass a bill to make Daylight Saving Time permanent. In a suprising-to-no-one twist, the bill is stalled in the House, where representatives are arguing over which of standard or daylight saving time should be permanent.

Would you prefer standard time or daylight saving time? If you’re not sure, check out the Daylight Saving Time Gripe Assistant Tool by Andy Woodruff on Observable.

I thought it’d be interesting to visualize day light hours for U.S. cities. You can use the dropdown below to choose your city or the nearest city with more than 100,000 residents (or just pick a random city!). Then, toggle between Standard or DST to see how either proposal would affect you. Or choose Both to see what will happen in the unimaginable case that the U.S. Congress doesn’t actually make DST permanent.

Pick a city in the United States ()

Use timezone

Standard

Both

DST

Sunrise and sunset times in Washington, D.C.

Inspiration

The layout for this vizualization was heavily inspired by a series by Kristina Szucs that I discovered via r/dataisbeautiful. I loved the aesthetic of Kristina’s plot and the subtle gradients and shadows of the daylight/twilight hour regions. Her attention to little details such a the sunrise and sunset icon labels and the stars in the night region are just fantastic.

How long are the nights in New York City? by Krisztina Szucs

In my version I wanted to draw on a similar structure and style to visualize daylight rather than sunset hours. I also wanted to see how far I could go with my plot without leaving the comfort of ggplot2, so I stopped short of adding the sunrise and sunset icons. (I suppose anything is possible in ggplot2 but IMHO this is a reasonable line to draw.)

Where are you?

To get accurate sunrise and sunset time data, we first need to figure out where in the world we are. Fortunately, the ipapi package makes it easy to grab key geolocation data from your IP address, like latitude, longitude and time zone. (I’m adding a little fuzz just to make ipapi a little less accurate.)

location <- as.list(ipapi::geolocate(NA, .progress = FALSE))

# find a location nearby but not my actual house, lol
location$lat <- location$lat + runif(1, min = -0.5, max = 0.5)
location$lon <- location$lon + runif(1, min = -0.5, max = 0.5)

location[c("lat", "lon", "timezone")]
## $lat
## [1] 33.34846
## 
## $lon
## [1] -85.17511
## 
## $timezone
## [1] "America/New_York"

Sunrise and Sunset Times

Next, we take our latitude and longitude to suncalc, an R package port of suncalc.js. We ask getSunlightTimes() for the dawn and dusk related times for every day in 2022.

sun_times <-
  suncalc::getSunlightTimes(
    date = seq(
      as.Date("2022-01-01"),
      as.Date("2023-01-01"),
      by = "day"
    ),
    lat = location$lat,
    lon = location$lon,
    tz = location$timezone,
    keep = c("dawn", "nauticalDawn", "dusk", "nauticalDusk", "sunrise", "sunset")
  )

head(sun_times)
##         date      lat       lon                dawn        nauticalDawn
## 1 2022-01-01 33.34846 -85.17511 2022-01-01 07:18:23 2022-01-01 06:47:27
## 2 2022-01-02 33.34846 -85.17511 2022-01-02 07:18:35 2022-01-02 06:47:41
## 3 2022-01-03 33.34846 -85.17511 2022-01-03 07:18:45 2022-01-03 06:47:53
## 4 2022-01-04 33.34846 -85.17511 2022-01-04 07:18:54 2022-01-04 06:48:03
## 5 2022-01-05 33.34846 -85.17511 2022-01-05 07:19:02 2022-01-05 06:48:12
##                  dusk        nauticalDusk             sunrise
## 1 2022-01-01 18:12:28 2022-01-01 18:43:24 2022-01-01 07:45:47
## 2 2022-01-02 18:13:11 2022-01-02 18:44:05 2022-01-02 07:45:57
## 3 2022-01-03 18:13:54 2022-01-03 18:44:47 2022-01-03 07:46:06
## 4 2022-01-04 18:14:39 2022-01-04 18:45:30 2022-01-04 07:46:14
## 5 2022-01-05 18:15:25 2022-01-05 18:46:14 2022-01-05 07:46:19
##                sunset
## 1 2022-01-01 17:45:04
## 2 2022-01-02 17:45:48
## 3 2022-01-03 17:46:33
## 4 2022-01-04 17:47:20
## 5 2022-01-05 17:48:07
##  [ reached 'max' / getOption("max.print") -- omitted 1 rows ]

If you’re curious about the difference between (civil) dawn, nautical dawn and sunrise, take a stroll through Twilight on Wikipedia.

Tidy Sun Times

As cool as it is to so easily get to the point of having this data in hand, we need to do a little bit of tidying up to get it ready for ggplot2. In particular, we need to consolidate all of the timestamps into one column that we can index by date and event (such as dawn, nautical dawn, etc.). I used pivot_longer() to move the column labels for dawn through sunset into an event column with the corresponding values from each row in an adjacent time column.

library(tidyverse)

tidy_sun_times <-
  sun_times %>%
  select(-lat, -lon) %>%
  pivot_longer(-date, names_to = "event", values_to = "time") %>%
  mutate(
    tz = strftime(time, "%Z"),
    time = hms::as_hms(time)
  )

tidy_sun_times
## # A tibble: 2,196 × 4
##    date       event        time     tz   
##                    
##  1 2022-01-01 dawn         07:18:23 EST  
##  2 2022-01-01 nauticalDawn 06:47:27 EST  
##  3 2022-01-01 dusk         18:12:28 EST  
##  4 2022-01-01 nauticalDusk 18:43:24 EST  
##  5 2022-01-01 sunrise      07:45:47 EST  
##  6 2022-01-01 sunset       17:45:04 EST  
##  7 2022-01-02 dawn         07:18:35 EST  
##  8 2022-01-02 nauticalDawn 06:47:41 EST  
##  9 2022-01-02 dusk         18:13:11 EST  
## 10 2022-01-02 nauticalDusk 18:44:05 EST  
## # … with 2,186 more rows

There’s also a small trick here to use strftime() to extract the short timezone label %Z, e.g. EST or EDT, for each day, which I’ll use later when calling out the time changes. And finally, I used hms::as_hms() to extract the time of day component from each sun event timestamp. The neat thing about the hms class is that while it prints in a readable hours, minutes, seconds format, we can also treat it as an integer number of seconds from midnight. We’ll use this property in just a bit when working on the plot’s axis labels.

First Looks

Now that we have tidy data ready for ggplot2, let’s plot it! This plot won’t look amazing, but it will help us get a sense of the data we have to work with.

ggplot(tidy_sun_times) +
  aes(x = date, y = time, color = event) +
  geom_line()

Paired Daily Events

The plot above reveals our next challenge: I used geom_line() to plot the time of each event as a line, but I want to be able to fill in the region between each pair of events:

sunrise and sunset
dawn and dusk
nautical dawn and nautical dusk.

We can use geom_ribbon() to acheive this look, but it requires a little more transformation. We need the sunrise time in one column called starts and a second column with the sunset time in ends. Then we can map these new columns to the ymin and ymax aesthetics, letting geom_ribbon() fill in space between them.

The plan of action is to create a new column that I’ll call period where we’ll choose which of these two new columns a timestamp will be moved to. We’ll identify each pair using the morning event label.

Because we need pivot_longer() to create two new columns, starts and ends, from our single period column, we’ll first split the table into a list of two tables, one for each period, and then pivot the timestamp column into a new column. Because we’re operating on a list, we’ll use purrr::map() to coordinate this action. Then we can merge the two tables back together with left_join(), using purrr::reduce() to apply that action to the list of pivoted tables.

After that, we’re back in single-table land, and can use mutate() and select() to tweak the final table output to make sure that the morning events are ordered correctly: nautical dawn, then dawn, then sunrise.

tidier_sun_times <-
  tidy_sun_times %>%
  mutate(
    period = case_when(
      str_detect(event, "[dD]awn|sunrise") ~ "starts",
      str_detect(event, "[dD]usk|sunset") ~ "ends"
    ),
    pair = recode(
      event,
      nauticalDusk = "nauticalDawn",
      sunset = "sunrise",
      dusk = "dawn"
    )
  ) %>%
  group_split(period) %>%
  map(pivot_wider, names_from = "period", values_from = "time") %>%
  reduce(
    left_join,
    by = c("date", "tz", "pair"),
    suffix = c("_ends", "_starts")
  ) %>%
  mutate(
    pair = factor(pair, c("nauticalDawn", "dawn", "sunrise"))
  ) %>%
  select(date, tz, pair, contains("starts"), contains("ends"))

tidier_sun_times
## # A tibble: 1,098 × 7
##    date       tz    pair         event_starts starts   event_ends   ends    
##                                       
##  1 2022-01-01 EST   dawn         dawn         07:18:23 dusk         18:12:28
##  2 2022-01-01 EST   nauticalDawn nauticalDawn 06:47:27 nauticalDusk 18:43:24
##  3 2022-01-01 EST   sunrise      sunrise      07:45:47 sunset       17:45:04
##  4 2022-01-02 EST   dawn         dawn         07:18:35 dusk         18:13:11
##  5 2022-01-02 EST   nauticalDawn nauticalDawn 06:47:41 nauticalDusk 18:44:05
##  6 2022-01-02 EST   sunrise      sunrise      07:45:57 sunset       17:45:48
##  7 2022-01-03 EST   dawn         dawn         07:18:45 dusk         18:13:54
##  8 2022-01-03 EST   nauticalDawn nauticalDawn 06:47:53 nauticalDusk 18:44:47
##  9 2022-01-03 EST   sunrise      sunrise      07:46:06 sunset       17:46:33
## 10 2022-01-04 EST   dawn         dawn         07:18:54 dusk         18:14:39
## # … with 1,088 more rows

Another plot

Now we’re ready to try out our even tidier data set! Okay, it’s actually less tidy, but more ready for plotting. We’ll swap out geom_line() for geom_ribbon(), and map starts to ymin and ends to ymax, filling in each region by pair.

ggplot(tidier_sun_times) +
  aes(date, ymin = starts, ymax = ends, fill = pair) +
  geom_ribbon()

It looks terrible, but it’s more or less what we want to see. The y-axis is a little confusing though because it reads from night at the top to day at the bottom. The trick is to recall that we used hms::as_hms() to turn the time of day into an integer number of seconds from midnight. So we can reverse the y-axis with scale_y_reverse() and then provide our own labels.

ggplot(tidier_sun_times) +
  aes(date, ymin = starts, ymax = ends, fill = pair) +
  geom_ribbon() +
  scale_y_reverse(
    limits = c(24*60^2, 0),
    breaks = seq(0, 24*60^2, by = 3 * 60^2),
    label = paste0(seq(0, 24, by = 3), ":00"),
    expand = expansion()
  ) +
  scale_x_date(
    breaks = "3 months",
    date_labels = "%b"
  )

Make it pretty

Great! Now it’s time to draw the rest of the owl! Which means I’m now going to include roughly 150 lines of code that take the rough sketch above and make it a pretty ggplot.

Of course I should mention that what you’ll find below isn’t even the full story of the plots you see in this post. It turned out to be an intermediate sketch of the code that I actually used to create the plots. It also turns out that I’m pretty good at writing code that creates more problems that I need to solve with more code.

A few preliminaries: I’ll use the Outfit font from Google Fonts with the help of the sysfonts package.

sysfonts::font_add_google("Outfit")

We’ll need a grid for the x- and y-axis that will be used in a few places, so I’ll create them up front. We end up with a vector of dates from January 1, 2022 to January 1, 2023 by 2 months for the x-axis and a vector of times from midnight to midnight by 3 hours for the y-axis.

x_breaks <- seq(
  from = as.Date("2022-01-01"),
  to = as.Date("2023-01-01"),
  by = "2 months"
)
y_breaks <- seq(0, 24*60^2, by = 3 * 60^2)

Finally, there are a couple of colors I used in more than one place for the foreground and background colors. There are a few other colors that I should have pulled into variables for clarity, but I’ve decided that it’s not worth the effort to think up variable names for them.

color_text <- "#F2CDB9"
color_bg <- "#39304a"

Finally, as promised, the 150ish lines of ggplot2 code. Enjoy!

ggplot(tidier_sun_times) +
  # The x-axis is always the day of the year
  aes(x = date) +
  # Behind everything we add a grid of `+` characters
  # in place of grid lines, to give a starry feel
  geom_text(
    # the data for this layer is our grid of x and y breaks
    data = cross_df(
      list(date = x_breaks, time = y_breaks, label = "+")
    ) %>%
      mutate(across(date, as.Date, origin = "1970-01-01")),
    aes(label = label, y = time),
    color = "#C29F5F"
  ) +
  # Here you'll recognize the outlines of our original plot sketch
  geom_ribbon(
    aes(ymin = starts, ymax = ends, fill = pair, alpha = pair),
    show.legend = FALSE
  ) +
  # Add dotted horizontal lines at 9am and 5pm
  geom_hline(
    yintercept = c(9, 17) * 60^2,
    color = color_bg,
    alpha = 0.5,
    linetype = 2
  ) +
  # And a little text to indicate the meaning of those lines
  annotate(
    geom = "text",
    x = min(tidier_sun_times$date),
    y = c(9, 17) * 60^2,
    label = c("9am", "5pm"),
    color = color_bg,
    hjust = -0.25,
    vjust = c(2, -1)
  ) +
  # If the timezone changes, it'll be due to daylight saving time
  # so we'll add a little text to highlight that change
  geom_text(
    # This combines my favorite magrittr pipe trick:
    #   `. %>%` creates a function with a single argument
    # with my favorite ggplot2 geom trik:
    #   `data` takes a function with a single argument that
    #   can be used to filter the global dataset to a smaller subset
    # The net result here is that we take the global data
    # and filter down to the two places where the timezone changes
    data = . %>%
      filter(tz != coalesce(lag(tz), first(tz))) %>%
      slice_head(n = 1),
    aes(y = ends, label = tz),
    hjust = 1,
    vjust = 1,
    nudge_x = -21,
    nudge_y = -60^2 * 1.5,
    lineheight = 0.8,
    color = color_text
  ) +
  # These add two little arrows to point from the timezone text
  # to the notch in the plot where the timezone change happens
  geom_curve(
    # Here's the `data = . %>%` trick again
    data = . %>%
      filter(pair == "nauticalDawn") %>%
      filter(tz != coalesce(lag(tz), first(tz))) %>%
      slice_head(n = 1),
    # This next bit took much fiddling.
    aes(
      x = date - 17,
      xend = date,
      y = ends - (-60^2 * 1.2),
      yend = ends + 500
    ),
    # If you like it put an arrow on it
    arrow = arrow(length = unit(0.08, "inch")),
    size = 0.5,
    color = color_text,
    curvature = 0.4
  ) +
  # The next two geoms highlight the second timezone change
  # and are copies of the previous two layers but use
  # `slice_tail()` instead of `slice_head()`.
  geom_text(
    data = . %>%
      filter(tz != coalesce(lag(tz), first(tz))) %>%
      slice_tail(n = 1),
    aes(y = starts, label = tz),
    hjust = 1,
    nudge_x = -21,
    nudge_y = 60^2 * 1.5,
    lineheight = 0.8,
    color = color_text
  ) +
  geom_curve(
    data = . %>%
      filter(pair == "nauticalDawn") %>%
      filter(tz != coalesce(lag(tz), first(tz))) %>%
      slice_tail(n = 1),
    aes(
      x = date - 17,
      xend = date,
      y = starts - 60^2,
      yend = starts - 500
    ),
    arrow = arrow(length = unit(0.08, "inch")),
    size = 0.5,
    color = color_text,
    curvature = -0.4
  ) +
  # Finally, we add a little annotation in the left edge of the plot
  # to serve as a legend for each layer and call out dawn, dusk,
  # sunrise and sunset, etc. Here I used the `ggrepel` package to
  # make sure the labels don't overlap, and in hopes that I wouldn't
  # need to fiddle too much with positioning. Fiddling was required
  # but I think the end result looks pretty good.
  ggrepel::geom_label_repel(
    data = . %>% filter(date == max(date)) %>%
      pivot_longer(contains("event")) %>%
      mutate(
        date = date + 12,
        time = if_else(value == pair, starts, ends),
        value = snakecase::to_title_case(value)
      ),
    aes(y = time, fill = pair, label = value),
    color = color_bg,
    fontface = "bold",
    show.legend = FALSE,
    # Most of the next few lines are designed to keep the
    # labels on the right side of the plot as close to the
    # layers they're supposed to annotate as possible.
    direction = "y",
    min.segment.length = 20,
    hjust = 0,
    label.size = 0,
    label.padding = 0.33,
    box.padding = 0.25,
    xlim = c(as.Date("2023-01-07"), NA)
  ) +
  # Next up, deal with our scales.
  # First up colors are the colors for the ribbon fill.
  scale_fill_manual(
    values = c(
      nauticalDawn = "#b56576",
      dawn = "#eaac8b",
      sunrise = "#ffd27d"
    )
  ) +
  # Then add a little opacity, even though ggplot will warn us
  # that using opacity with a discrete variable isn't a good idea.
  # (I think it's a fine idea, thank you very much.)
  scale_alpha_discrete(range = c(0.5, 0.9)) +
  # Here are the x- and y-axis scales from our original sketch
  scale_x_date(
    breaks = x_breaks,
    date_labels = "%b",
    limits = c(
      as.Date("2022-01-01"),
      as.Date("2023-03-15")
    ),
    expand = expansion()
  ) +
  scale_y_reverse(
    limits = c(
      max(tidier_sun_times$ends + 60^2),
      min(tidier_sun_times$starts - 60^2)
    ),
    breaks = y_breaks,
    labels = paste0(seq(0, 24, by = 3), ":00"),
    expand = expansion()
  ) +
  # Labels, obvs.
  labs(
    x = NULL,
    y = NULL,
    title = "How long are the days near me?",
    subtitle = "Atlanta, GA",
    caption = "garrickadenbuie.com"
  ) +
  # Make sure the sunrise/sunset labels aren't clipped by the plot area
  coord_cartesian(clip = "off") +
  # Finally, make it pretty. We'll start with a minimal base theme
  theme_minimal(base_family = "Outfit", base_size = 16) +
  # And then tweak a bunch of little things...
  theme(
    plot.title = element_text(
      color = color_text,
      hjust = 0,
      size = 14
    ),
    plot.subtitle = element_text(
      color = color_text,
      hjust = 0,
      size = 24,
      margin = margin(b = 6)
    ),
    plot.title.position = "plot",
    plot.background = element_rect(fill = color_bg),
    plot.margin = margin(20, 0, 20, 10),
    panel.grid = element_blank(),
    axis.text = element_text(color = color_text),
    axis.title = element_text(color = color_text),
    plot.caption = element_text(
      color = "#726194",
      hjust = 0.97,
      vjust = -1
    ),
    plot.caption.position = "plot"
  )

Wordle Guess Helper

Garrick Aden-Buie — Mon, 21 Feb 2022 05:00:00 GMT

Have you heard of Wordle? Who am I kidding, of course you’ve heard of Wordle! In fact, I’m pretty certain we’re way past peak Wordle at this point.

Here’s a Wordle helper that doesn’t completely take the fun out of the guessing, while also making sure you’ve got a good chance at winning every time.

Type your guesses below and then use the buttons below each letter to report Wordle’s response. Press return to start the next guess. Use delete to remove letters or words you’ve entered.

As soon as you add the results for a new word, the table of next guess candidates will update! Pick wisely.

Intro

I started this post about a month ago, roughly at the same time that every other person with a blog about doing things with computers also decided to start writing a post about Wordle.

I’ve been tempted to just walk away from this post more than once. After all, since I’ve started writing this post Wordle has been solved more than once, Winston Chang rewrote Wordle in Shiny, roughly 70 other people wrote Wordle clones or helper apps and packages in R alone. Felienne Hermans wrote a Twitter bot to guess the word from shared game emojis. Someone else wrote a bot to intentionally ruin everyone’s day by spoling the answer to the next day’s Wordle. (Both bots were eventually suspended by Twitter.) Oh and Wordle was bought for big money by the New York Times who fumbled the handoff and lost more than a few player’s word streaks in the transfer.

I should admit up-front that I’ve never really played Wordle. It’s exactly the kind of task that immediately cries out to be automated: I’d apparently much rather spend a month’s worth of after-hours tinkering time to think through and codify a decent strategy than to just think up some words on my own.

And yet I love Wordle. I think it’s awesome. The rules are simple, but deceptively ambiguous. The game play is so concise it can fit in a tweet (even though that’s annoying for accessibility reasons). Still, the UI is simple, intuitive and fun without trying to hack your brain to be addictive. It’s a feel good game.

Another reason to love Wordle: there are so many great programming tasks around Wordle. It’s easy to describe the mechanics, to understand the game play, to look at the app and think: I can do that. Which is why, right now, programmers are hard at work tinkering over word lists or practicing web development in their favorite framework using Wordle.

As an educator, it means you can tailor a Wordle-based programming challenge to be as simple or complicated as needed. Once you start to break down the game, it’s more complicated than it appears at first glance, and there’s so much to choose from. State management, data structures, browser storage, game theory, CSS, user interface design, accessibility. You can go deep on any of these topics.

So if you have a Wordle idea you want to tinker with, I wholeheartedly encourage you to run with it. Let Wordle inspire you to practice using regular expressions with stringr, web scraping with httr, text processing with Python, working with Twitter data with rtweet, or making accessible plots with ggplot2.

What follows here is a bit of a journey. It is not the best strategy for Wordle or even the best way to play. But along the way we’ll learn a few text processing tricks, we’ll write a few functions, and we’ll learn how to move seamlessly from R to the browser in the same document or blog post. (The R code and data I write below create the word data used in the table and app above!)

Let’s look at some words

Let’s dig in. To get started, I’m using a few of the usual suspects from the tidyverse package. Out of habit, I’ll load the ones I want specifically. (I think I also used tidyr somewhere in here, too.)

library(dplyr)
library(purrr)
library(stringr)

Now we’re ready to load our word list. At first I started with Scrabble’s word list, but it turns out that Wordle included the complete word list in its source code. (You could call it a hack but only in the state of Missouri.)

I used my elite hacker copy-and-paste skills to store Wordle’s word list as a JSON file (165K).

wordle_words <- jsonlite::fromJSON(
  "wordle.json",
  simplifyVector = TRUE
)

It turns out that Wordle maintains two separate lists. One list contains the 2,315 words used as solutions

sample(wordle_words$answers, 5)

[1] "exert" "petty" "scent" "fishy" "askew"

and the other contains the 10,657 words that the game considers a valid guess.

sample(wordle_words$words, 5)

[1] "vires" "neese" "koppa" "huhus" "ryked"

Do the two word lists overlap?

wordle_words %>%
  reduce(intersect) %>%
  length()

[1] 0

No, they do not (the intersection of the two word lists is empty). We could make things super easy for ourselves by only considering the words on the solution list, but that would really ruin the fun. So let’s combine the two lists.

words <- unlist(wordle_words)

Now lets turn those words into data we can work with.

A letter popularity contest

Popularity by word

My first thought (and I think it’s many people first thought) was to consider the probability that a letter appears in a word. In other words: does R appear in more words than F?

To answer this we can split each word into a vector of letters, take only the unique letters, and then count how many times each letter appears in a word.

Splitting the word into a vector of letters is something we’ll be doing a lot, and stringr::str_split() or strsplit() can help. The trick is to use an empty string as the split pattern to break apart each string character by character.

str_split(c("unhip", "jeans"), "")

[[1]]
[1] "u" "n" "h" "i" "p"

[[2]]
[1] "j" "e" "a" "n" "s"

Note that this process takes our vector and gives us a list of vectors, which means we’ll be seeing a lot of purrr’s map() function in this post.

letter_freq <-
  words %>%
  # Split each word into a vector of letters
  str_split("") %>%
  # Keep one of each letter per word
  map(unique) %>%
  # Unlist into a big vector of letters
  unlist() %>%
  # Count the letters (each appearance in a word)
  table() %>%
  # Most popular letters first
  sort(decreasing = TRUE) %>%
  # Turn into frequency table
  `/`(length(words)) %>%
  # Remove attributes from table()
  c()

letter_freq

          s           e           a           o           r           i 
0.457600987 0.439793401 0.410884983 0.301495529 0.301341351 0.276672834 
          l           t           n           u           d           y 
0.240055504 0.233811286 0.214847364 0.187789084 0.177150786 0.156567993 
          c           p           m           h           g           b 
0.148011101 0.145312982 0.144002467 0.131668208 0.118948504 0.117098366 
          k           w           f           v           z           j 
0.111316682 0.079247610 0.076318224 0.051958064 0.030141844 0.022278754 
          x           q 
0.022124576 0.008556892

Note that we only counted each letter once per word, so we now know that R appears in 30% of the words in the word list, while F appears in only 8%. A first guess that includes R would probably be better than one with an F.

Popularity by position

Another way to look at letter frequency would be to consider the position of the letter in the word. What if we know that R and F are in the word: which is a more likely choice as the fourth letter?

To do this we…

First turn the word list into a tibble with one row per word.
Then, using tidyr::separate_rows(), we can add a new column with the letters in each word.
Grouping by word and adding a row_number() gives us the position of each letter in the word.
Then we can count the number of times each letter occurs in a given position with a new group_by() and summarize() (we could have used count() with another ungroup(), too).
Then, if we re-use our letter-word counts from the last step, we can count the number of words that have a the letter in question so that our frequency is effectively given the letter R, how often does it appear as the fourth letter?
Finally, tidyr::pivot_wider() moves the positions to the columns so the table is easier to read.

letter_freq_pos <-
  tibble(word = words) %>%
  select(word) %>%
  mutate(letter = word) %>%
  tidyr::separate_rows(letter, sep = "") %>%
  filter(letter != "") %>%
  group_by(word) %>%
  mutate(position = row_number()) %>%
  group_by(letter, position) %>%
  summarize(n = n(), .groups = "drop") %>%
  mutate(
    words = letter_freq[letter] * length(!!words),
    freq = n / words
  ) %>%
  select(-n, -words) %>%
  tidyr::pivot_wider(
    names_from = position,
    values_from = freq,
    values_fill = 0,
    names_prefix = "p"
  )

letter_freq_pos

# A tibble: 26 × 6
   letter     p1     p2     p3    p4     p5
             
 1 a      0.138  0.425  0.232  0.202 0.128 
 2 b      0.598  0.0533 0.221  0.160 0.0388
 3 c      0.480  0.0917 0.204  0.214 0.0661
 4 d      0.298  0.0366 0.170  0.205 0.358 
 5 e      0.0531 0.285  0.155  0.408 0.267 
 6 f      0.604  0.0242 0.180  0.235 0.0828
 7 g      0.413  0.0493 0.236  0.274 0.0927
 8 h      0.286  0.320  0.0703 0.138 0.217 
 9 i      0.0460 0.385  0.293  0.245 0.0780
10 j      0.699  0.0381 0.159  0.100 0.0104
# ℹ 16 more rows

Now we can answer our question about R and F in the fourth position.

letter_freq_pos %>%
  filter(letter %in% c("r", 'f')) %>%
  select(letter, p4)

# A tibble: 2 × 2
  letter    p4
    
1 f      0.235
2 r      0.184

So R is the fourth letter in 18% of the words containing R — but F is the fourth letter in 24% of its words.

Ideally this information will help us filter guesses when we know that a set of letters are in the solution, but we don’t yet know where.

First Choice

What word should we guess first? Ideally, we want a word whose answer gives us the most information. Intuitively, if we pick a word that has the most popular letters and each letter is different, we’ll be able to discard or include the most words when Wordle tells us which letters are in or out.

Formally, this calculation is called entropy. It measures how much information is contained in a particular instance of a random process. In this case, words with higher entropy give us more information because they encode more information.

This is all a little hand-wavy, so I’ll just duck the details and call this number a score. The higher the score, the better the word choice.

To calculate the entropy score, we take a word, split it into it’s letters, and then get the probability of each letter appearing in the word. Duplicated letters don’t tell us much, so we set second appearances of a letter close to zero. And then we calculate entropy

which in R code is

- sum(p * log(p, base = 2))

where p is a vector of probabilities for a given outome.

We can wrap all of that up into a function score_entropy():

score_entropy <- function(word) {
  chars <- str_split(word, "")[[1]]
  p <- letter_freq[chars]
  # we learn something but not much from duplicated letters
  p[duplicated(chars)] <- min(letter_freq)
  - sum(p * log(p, base = 2))
}

Notice that score_entropy() isn’t vectorized, so we’ll have to use a map() function to call it over a vector of words. We can be even more specific and use map_dbl() since we know that score_entropy() returns a number.

c("unhip", "jeans", "pools") %>%
  set_names() %>%
  map_dbl(score_entropy)

   unhip    jeans    pools 
2.232150 2.163479 1.994939

This tells us, broadly, that unhip is a better choice than jeans and pools is worse than either. (Intuitively: you don’t learn much from the second O.)

Let’s use this to create a table of words and their associated entropy scores. Taking a peek at the highest scoring words tells us…

words_first_choice <-
  tibble(word = words) %>%
  mutate(score = map_dbl(word, score_entropy)) %>%
  arrange(desc(score))

words_first_choice

# A tibble: 12,972 × 2
   word  score
    
 1 arose  2.61
 2 aeros  2.61
 3 soare  2.61
 4 arise  2.60
 5 raise  2.60
 6 aesir  2.60
 7 reais  2.60
 8 serai  2.60
 9 osier  2.59
10 realo  2.59
# ℹ 12,962 more rows

… that according to this measure, the best first-choice words are arose, aeros, and soare. arose uses all five of the letters that most commonly appear in a word, and is also probably (okay, it is) on the answers list, so hello new first word choice!

Second Choice

After your first choice, you know up to three pieces of additional information. Some of the letters in your guess

dark square aren’t in the solution
yellow square are in the solution but not where you guessed
green square are in the solution and are where you guessed

None of the letters are in the solution

What if you guessed arose and got five gray boxes telling you that none of those letters appear in the solution?

a r o s e

We need to discard any words with a, r, o, s, or E in them. To do this, we’ll write a small function str_has_none_of() that takes a vector of words and a vector of letters, and checks if any of the letters are in each of the words. Technically, we use our same str_split() trick to split each word into a vector of letters and then check that the intersection of word letters and unwanted letters is empty.

str_has_none_of <- function(words, letters) {
  words <- str_split(words, "")
  map_lgl(words, ~ length(intersect(letters, .x)) == 0)
}

Using this function, we can quickly reduce our word list from 12,972 to 577 words.

words_first_choice %>%
  filter(str_has_none_of(word, c("a", "r", "o", "s", "e")))

# A tibble: 577 × 2
   word  score
    
 1 unlit  2.43
 2 until  2.43
 3 linty  2.39
 4 clint  2.38
 5 unlid  2.38
 6 culti  2.36
 7 tulip  2.35
 8 uplit  2.35
 9 unity  2.35
10 lindy  2.34
# ℹ 567 more rows

This new word list suggests that unlit or until would be a good next choice, so we’ll go with until. And if none of the letters in arose and until appear in the solution…

a r o s e

u n t i l

letters_guess <- str_split("arose until", "")[[1]]

words_first_choice %>%
  filter(str_has_none_of(word, letters_guess))

# A tibble: 3 × 2
  word  score
   
1 pygmy  1.65
2 hyphy  1.33
3 gyppy  1.31

then your answer is most definitely one of pygmy, hyphy, or gyppy.

Right letter, wrong place

If you learn something from the guess, though, you can filter the word list based on the information you just learned.

Say we guess arose and wordle reveals that R and O appear in the solution.

a r o s e

We now know that the solution:

Doesn’t have A, S or E
Does contain R and O
Doesn’t have R as the 2^nd letter or O as the 3^rd.

We’ve already implemented this the first step by discarding words with str_has_none_of(). We also need a similar version called str_has_all_of() to keep only words that have letters we know are in the solution.

str_has_all_of <- function(words, letters) {
  words <- str_split(words, "")
  map_lgl(words, ~ length(setdiff(letters, .x)) == 0)
}

str_has_all_of("rhino", c("r", "o"))

[1] TRUE

And finally we can use regular expressions to keep track of the third piece of information:

.[^r][^o]..

A . means any letter at that spot in the word (other than the ones we’ve excluded). The [] indicate a set of options that could be present at a location in the string. The opening ^ negates the selection, so [^r] means a character that isn’t r.

words_first_choice %>%
  filter(
    str_has_none_of(word, c("a", "s", "e")),
    str_has_all_of(word, c("r", "o")),
    str_detect(word, ".[^r][^o]..")
  )

# A tibble: 142 × 2
   word  score
    
 1 lirot  2.54
 2 intro  2.52
 3 nitro  2.52
 4 nidor  2.47
 5 roily  2.47
 6 loric  2.46
 7 toric  2.45
 8 milor  2.45
 9 corni  2.44
10 porin  2.44
# ℹ 132 more rows

lirot is an unusual word, so let’s choose the next word on the list: intro.

a r o s e

i n t r o

Wordle thinks and tells us that we have T in the right spot! Also, we now know that I and N aren’t in the solution, and we still haven’t got R and O in the right place.

Right letter, right place

We can repeat the step above, but using a new regular expression:

.[^r]t[^ro][^o]

Notice that we know a little more about where R and O can’t be, but importantly the t in the middle letter ensures we find words with T in the right place.

This leaves us with a few good choices:

words_first_choice %>%
  filter(
    str_has_none_of(word, c("a", "s", "e", "i", "n")),
    str_has_all_of(word, c("r", "o", "t")),
    str_detect(word, ".[^r]t[^r][^o]")
  )

# A tibble: 4 × 2
  word  score
   
1 rotch  2.33
2 tutor  2.05
3 motor  1.99
4 rotor  1.65

rotch seems very unlikely, so we can pick from tutor, motor and rotor. But notice that the these include a small set of the same letters. In a sense, we might ask ourselves a new question — which is the more likely starting combination: tu, mo or ro?

At this point, you could just guess. It is a game after all! But no, let’s power forward and add more complexity to this blog post.

What if we switched our scoring at this point and considered the position of the letters in the candidate words? Doing something medium-naive, let’s frame this as: what’s the probability of T in the first position and U in the second and so on…

score_by_position <- function(word) {
  chars <- str_split(word, "")[[1]]

  res <- c()
  for (i in seq_along(chars)) {
    pos_alpha <- which(letters == chars[i])
    p <- letter_freq_pos[[str_c("p", i)]][pos_alpha]
    res <- c(res, p)
  }

  prod(res)
}

words_score_pos <-
  tibble(word = words) %>%
  mutate(
    score_pos = map_dbl(word, score_by_position),
    score_pos = score_pos / diff(range(score_pos))
  ) %>%
  arrange(desc(score_pos))

words_score_pos

# A tibble: 12,972 × 2
   word  score_pos
        
 1 foxes     1.00 
 2 boxes     0.991
 3 jones     0.864
 4 juves     0.808
 5 coxes     0.795
 6 faxes     0.792
 7 poxes     0.754
 8 fones     0.746
 9 bones     0.739
10 fixes     0.719
# ℹ 12,962 more rows

If we join this with our “new information” score, we now have to scores to choose from:

words_first_choice %>%
  filter(
    str_has_none_of(word, c("a", "s", "e", "i", "n")),
    str_has_all_of(word, c("r", "o", "t")),
    str_detect(word, ".[^r]t[^r][^o]")
  ) %>%
  left_join(words_score_pos) %>%
  arrange(desc(score_pos))

# A tibble: 4 × 3
  word  score score_pos
        
1 motor  1.99    0.0304
2 tutor  2.05    0.0200
3 rotch  2.33    0.0199
4 rotor  1.65    0.0132

Now we see that motor and tutor are the most likely words based on their position. We guess motor… and we’re right!

a r o s e

i n t r o

m o t o r

It only took three guesses! It’s almost like I planned this example to work out just like I wanted.

Generalizing

Okay, let’s do this for any number of guesses. First, let’s join our scored words into a single data frame.

words_scored <-
  left_join(
    words_first_choice,
    words_score_pos,
    by = "word"
  )

Then, we need a function that takes our guesses and results and generalizes them into the pieces of information our guesses reveal about the solution. This function is going to take a vector of guesses and a vector of results. The guesses are just the words we guessed, but we’ll need to invent a syntax to concicesly report the results. Here’s the syntax I decided to use:

. means the letter is absent
- means the letter is present (wrong place)
+ means the letter is correct (right place)

In broad strokes, the function will take each guess and use the result

Pull out the correct letters and their positions in exact so we can pick out words with letters in those spots.
Pull out present letters and their positions into exclude so we can compose the regular expression to filter out words that have these letters in those places.
Add the present by wrong place letters to bucket_keep, a bucket of letters that we know are in the solution.
And add any absent letters to bucket_dicard so we can filter out words that have any of these letters.
The last step is to compose the regular expression pattern from exact and exclude, and then return the regexp and the letters to keep and discard.

#' @param guesses A vector of words that you have guessed
#' @param result A vector of results for each guess using `.` for a miss, `-`
#'   for a letter in the solution that isn't in the right place and `+` for a
#'   letter that's in the right spot.
summarize_guesses <- function(guesses, results) {
  stopifnot(all(str_length(c(guesses, results)) == 5))

  guesses <- str_split(guesses, "")
  results <- str_split(results, "")

  exclude <- character(5)
  exact <- character(5)
  bucket_keep <- c()
  bucket_discard <- c()

  for (i in seq_along(guesses)) {
    g <- guesses[[i]]
    r <- results[[i]]

    if (any(r == "+")) {
      exact[r == "+"] <- g[r == "+"]
      bucket_keep <<- c(bucket_keep, g[r == "+"])
    }
    if (any(r == "-")) {
      bucket_keep <- c(bucket_keep, g[r == "-"])
      exclude[r == "-"] <- paste0(exclude[r == "-"], g[r == "-"])
    }
    if (any(r == ".")) {
      bucket_discard <- c(bucket_discard, g[r == "."])
    }
  }

  exclude[exclude != ""] <- paste0("[^", exclude[exclude != ""], "]")
  exact[exact == ""] <- NA_character_
  exclude[exclude == ""] <- NA_character_

  pattern <- coalesce(coalesce(exact, exclude), ".")

  # Say you guess a word with two Ts,
  # but there's only one T in the solution.
  # T will appear on keep and discard bucket,
  # so we need to explicitly keep it.
  # (we could use that info, though, e.g. at most 1 T)
  bucket_discard <- setdiff(bucket_discard, bucket_keep)

  list(
    discard = unique(bucket_discard),
    keep = unique(bucket_keep),
    pattern = str_c(pattern, collapse = "")
  )
}

Remember when we guessed arose and got this result?

a r o s e

Our new function summarizes the information we’ve learned from this guess.

summarize_guesses(
  guesses = "arose",
  results = ".--.."
)

$discard
[1] "a" "s" "e"

$keep
[1] "r" "o"

$pattern
[1] ".[^r][^o].."

Then we guessed intro and got this result.

a r o s e

i n t r o

And again we have this summary.

guess_results <-
  summarize_guesses(
    guesses = c("arose", "intro"),
    results = c(".--..", "..+--")
  )

guess_results

$discard
[1] "a" "s" "e" "i" "n"

$keep
[1] "r" "o"

$pattern
[1] ".[^r]t[^r][^o]"

To get the remaining possible words, we can use this information to filter down to the words that

have none of the $discard letters
have all of the $keep letters
match the regular expression $pattern.

words_scored %>%
  filter(
    str_has_none_of(word, guess_results$discard),
    str_has_all_of(word, guess_results$keep),
    str_detect(word, guess_results$pattern)
  )

# A tibble: 4 × 3
  word  score score_pos
        
1 rotch  2.33    0.0199
2 tutor  2.05    0.0200
3 motor  1.99    0.0304
4 rotor  1.65    0.0132

All together now

Now that we know how to summarize and use the guess results to filter our next word choices, we can do this in one step with another small function, score_next_guess().

score_next_guess <- function(guesses, results) {
  guess_results <- summarize_guesses(guesses, results)

  words_scored %>%
    filter(
      str_has_none_of(word, guess_results$discard),
      str_has_all_of(word, guess_results$keep),
      str_detect(word, guess_results$pattern)
    )
}

Having guessed arose and intro, what would happen if we guessed rotch¹ next?

a r o s e

i n t r o

r o t c h

score_next_guess(
  guesses = c("arose", "intro", "rotch"),
  results = c(".--..", "..+--", "-++..")
)

# A tibble: 1 × 3
  word  score score_pos
        
1 motor  1.99    0.0304

From rotch we learn that the first letter isn’t R, but the second letter is o, which leaves us just one choice: motor.

m o t o r

Guessing Wordle words in real life

Beginner’s Luck

I wrapped up the score_next_guess() function on January 16th, 2022, which happened to be the easiest Wordle day of any day I’ve “played”. But it was a nice motivator to feel like I had spent my Sunday tinkering time well.

Opening with arose lead to a pleasant surprise.

a r o s e

From 12,972 words down to 37 words with our first guess. Nice!

# 2022-01-16
score_next_guess(
  guesses = c("arose"),
  results = c("----.")
)

# A tibble: 37 × 3
   word  score score_pos
         
 1 solar  2.58    0.0327
 2 soral  2.58    0.0327
 3 ratos  2.58    0.0404
 4 rotas  2.58    0.0576
 5 sorta  2.58    0.0401
 6 taros  2.58    0.102 
 7 toras  2.58    0.145 
 8 sonar  2.56    0.0416
 9 roans  2.56    0.0923
10 sarod  2.53    0.0537
# ℹ 27 more rows

Let’s just pick the first word on the list: solar.

a r o s e

s o l a r

Very nice!

Problematic words

In working on this, I ran into more than a few posts that had trouble with a few more obscure words, like igloo and ferry.

igloo

How many guesses would it take for us to get to igloo?

Round 1

a r o s e

Opening with arose is helpfulish.

score_next_guess(
  guesses = c("arose"),
  results = c("..-..")
)

# A tibble: 463 × 3
   word  score score_pos
         
 1 doilt  2.46  0.0680  
 2 indol  2.45  0.000646
 3 tondi  2.44  0.0195  
 4 lotic  2.43  0.00802 
 5 noily  2.42  0.0711  
 6 pilot  2.42  0.0501  
 7 colin  2.41  0.0801  
 8 nicol  2.41  0.00613 
 9 tonic  2.41  0.0198  
10 ontic  2.41  0.000670
# ℹ 453 more rows

Many of the words are obviously not the answer. Pilot is the first reasonable word on the list, and its score is relatively similar to the other top word choices, so I’d go with pilot.

Round 2

a r o s e

p i l o t

Picking pilot is a good choice!

score_next_guess(
  guesses = c("arose", "pilot"),
  results = c("..-..", ".-++.")
)

# A tibble: 1 × 3
  word  score score_pos
        
1 igloo  1.95  0.000268

Round 3

a r o s e

p i l o t

i g l o o

:tada: Great work!

ferry

Apparently there was a general furor about ferry when it was the Wordle solution of the day. Let’s see how long it takes us to get to that word.

Round 1

a r o s e

Opening with arose narrows down our word choices to 357 words.

score_next_guess(
  guesses = c("arose"),
  results = c(".-..-")
)

# A tibble: 357 × 3
   word  score score_pos
         
 1 liter  2.54  0.0250  
 2 relit  2.54  0.0180  
 3 tiler  2.54  0.0485  
 4 liner  2.53  0.0425  
 5 inert  2.52  0.000951
 6 inter  2.52  0.00199 
 7 niter  2.52  0.0157  
 8 uteri  2.50  0.000332
 9 idler  2.49  0.000788
10 riled  2.49  0.0604  
# ℹ 347 more rows

liter is both a word and at the top of our list, so it’s an easy next choice.

Round 2

a r o s e

l i t e r

The word list is now full of words with similar patterns, so let’s sort by position score to help us choose.

score_next_guess(
  guesses = c("arose", "liter"),
  results = c(".-..-", "...--")
) %>%
  arrange(desc(score_pos))

# A tibble: 50 × 3
   word  score score_pos
         
 1 jerky  1.94     0.334
 2 ferny  2.22     0.235
 3 perky  2.22     0.218
 4 jerry  1.64     0.177
 5 query  1.97     0.153
 6 ferry  1.80     0.153
 7 berry  1.88     0.151
 8 pervy  2.09     0.145
 9 perdy  2.31     0.128
10 kerky  1.87     0.125
# ℹ 40 more rows

Round 3

a r o s e

l i t e r

j e r k y

Now we’re down to 17 words to choose from. Still complicated. But if we arrange by position score, our top two choices are ferny and ferry.

You can see where this is headed, but let’s pretend we had no idea. Which would you pick?

score_next_guess(
  guesses = c("arose", "liter", "jerky"),
  results = c(".-..-", "...--", ".++.+")
) %>%
  arrange(desc(score_pos))

# A tibble: 17 × 3
   word  score score_pos
         
 1 ferny  2.22    0.235 
 2 ferry  1.80    0.153 
 3 berry  1.88    0.151 
 4 pervy  2.09    0.145 
 5 perdy  2.31    0.128 
 6 germy  2.23    0.122 
 7 derny  2.38    0.116 
 8 perry  1.92    0.115 
 9 mercy  2.27    0.109 
10 merry  1.92    0.0937
11 verry  1.74    0.0907
12 derry  1.96    0.0753
13 herry  1.91    0.0723
14 derby  2.27    0.0655
15 herby  2.21    0.0629
16 nervy  2.16    0.0371
17 nerdy  2.38    0.0328

Round 4

a r o s e

l i t e r

j e r k y

f e r n y

Now we’re down to 1 words to choose from.

score_next_guess(
  guesses = c("arose", "liter", "jerky", "ferny"),
  results = c(".-..-", "...--", ".++.+", "+++.+")
) %>%
  arrange(desc(score_pos))

# A tibble: 1 × 3
  word  score score_pos
        
1 ferry  1.80     0.153

Round 5

a r o s e

l i t e r

j e r k y

f e r n y

f e r r y

:tada: We did it! 5 isn’t bad, especially considering the terrible choices we had in round 3.

Make it an app

It’s awesome being able to run R code to test things out, but it’s also a little tedious. Since we’ve done the heavy lifting of prepping and scoring words, it’d be great if we could have a little web app that would help us

Input our guesses and results
Show us possible words after each round

And since I’m writing this blog post in R Markdown via blogdown, I can do it all right here!

Move the data from R to the web

The first thing we need to do is save our data in a way that it can be accessed by JavaScript in the browser. To do this, we’ll take our words_scored table and use jsonlite::write_json() to save the data frame as JSON.

words_scored %>%
  mutate(across(starts_with("score"), round, digits = 2)) %>%
  jsonlite::write_json("wordle-scored.json")

Now we have the data in a JSON file (that you can download if you want).

But to make life even easier, I’m going to use a trick I learned from htmlwidgets. What we can do is embed in the JSON file, which is only 589K, in a

Now that we have the data in a place where we can get it, let’s switch gears and write some JavaScript!



Start working in JavaScript
Here’s the cool thing: from here on out, the actual computation of the rest of the blog post is done in your browser. To facilitate this, I’ll use an extension I built for knitr for literate JavaScript programming with the js4shiny package.

js4shiny
Setting up literate JavaScript in blogdown is pretty straight-forward thanks to a little helper function from js4shiny.

js4shiny::html_setup_blogdown(stylize = "none")






tidyjs
The other cool thing we’ll use is tidyjs. It’s a really neat JavaScript library that makes it easy to work with data frames in the browser. If you squint really hard, it’s remarkably similar to the tidyverse, just with a JavaScript spin.
I wrapped tidyjs in an R package that automatically stays up to date with the latest version of tidyjs. To use tidyjs, we just need to call use_tidyjs().

tidyjs::use_tidyjs()




Now that we’ve included tidyjs in this page, we can finally switch to writing JavaScript instead of R.
First, we need to import a couple of functions from tidyjs that we’re going to want to use. With tidyjs, all transformations are wrapped in a call tidy(), so we have to import tidy. We also need filter() and sliceMax() for easy filtering.

const { tidy, filter, sliceMax } = Tidy




Load our data
The next step is to find the JSON data that we just serialized and stashed in our page. We can use document.getElementById() to find the element with the id 'words-scored', and then grab the JSON text itself from the .innerText property of that object. Finally, we call JSON.parse() on the json text to parse it into a JavaScript object.

wordsScored = JSON.parse(
  document.getElementById('words-scored').innerText
)




Preview the data
Here’s a quick preview of the data. In tidyjs you wrap a pipeline in tidy() and then each additional argument to tidy() is the next step in the pipe. To make it look a little more familiar to R users, I’ve added the %>% in the comments.

tidy(
  wordsScored, // %>%
  sliceMax(5, 'score')
)








Same song, different dance

Summarizing guesses
Next, we translate summarize_guesses() from R to summarizeGuesses() in JavaScript.

function summarizeGuesses ({ guesses, results }) {
  // Check that all guesses and results have 5 characters
  const allComplete = [...guesses, ...results].every(s => s.length == 5)
  if (!allComplete) {
    console.error('All guesses and results must have 5 characters.')
    return
  }

  // R: str_split(x, '')
  guesses = guesses.map(s => s.split(''))
  results = results.map(s => s.split(''))

  let exclude = Array(5).fill('')
  let exact = Array(5).fill('')
  let keep = []
  let discard = []

  for (i = 0; i < guesses.length; i++) {
    let g = guesses[i] // g: an array of 5 letters of a guess
    let r = results[i] // r: an array of 5 letters of the result

    for (j = 0; j < r.length; j++) {
      if (r[j] == '+') {
        // this letter is exactly right
        exact[j] = g[j]
        keep.push(g[j])
      } else if (r[j] == '-') {
        // this letter is included, wrong place
        keep.push(g[j])
        // so exclude it from this position
        exclude[j] += g[j]
      } else {
        // this letter isn't in the solution
        discard.push(g[j])
      }
    }
  }

  // build up the regex pattern blending `exact` and `exclude`
  const pattern = Array(5).fill('.')
  for (i = 0; i < 5; i++) {
    if (exact[i] != '') {
      pattern[i] = exact[i]
    } else if (exclude[i] != '') {
      pattern[i] = `[^${exclude[i]}]`
    }
  }

  discard = discard.filter(x => !keep.includes(x))
  return {discard, keep, pattern: pattern.join('')}
}


Here’s a quick preview of summarizeGuesses().

let summary = summarizeGuesses({
  guesses: ["arose", "indol"],
  results: ["..-..", "+..+-"]
})
console.log(summary)







Searching for the next word
And then we need to do the same for score_next_guess(). Of course, at this point I’m older and wiser and choose a better name: searchNextGuess().

function searchNextGuess ({ guesses, results }) {
  const guessResult = summarizeGuesses({guesses, results})

  return tidy(
    wordsScored,
    // discard words that contain a letter in the discard pile
    filter(d => !guessResult.discard.some(l => d.word.includes(l))),
    // keep only words that have all letters in the keep pile
    filter(d => guessResult.keep.every(l => d.word.includes(l))),
    // keep words that are consistent with results to date
    filter(d => RegExp(guessResult.pattern).test(d.word))
  )
}


Let’s prove to ourselves that these functions work.

let next = searchNextGuess({
  guesses: ["arose", "indol"],
  results: ["..-..", "+..+-"]
})

console.log(`There is ${next.length} word available for our next guess:`)
console.table(next[0])





Let’s try again. What if we chose a different second guess?

let rounds = {
  guesses: ["arose", "intro"],
  results: [".--..", "..+--"]
}
let next = searchNextGuess(rounds)

console.log('Guess summary ----')
console.log(summarizeGuesses(rounds))

console.log('Next word choices ----')
next.forEach(ws => console.log(`${ws.word} (${ws.score})`))








Now build the rest of the owl
Okay, this is the point where I confess that I went way off-track in building the little app at the top of this post. I fully intended to write about that part too, but honestly I’ve done a good job curing myself of the Wordle bug with this post.
For the curious, all the JavaScript code for the guess helper lives in wordle-component.js. Or, right click on this page and pick Inspect Element and find your way to the Sources or Debugger tab for a better look. It’s all vanilla JavaScript.
Also a quick shout-out to gridjs, which turned out to be a very easy way to create the table of sorted words.
<script src="https://unpkg.com/gridjs/dist/gridjs.umd.js">script>
<link href="https://unpkg.com/gridjs/dist/theme/mermaid.min.css" rel="stylesheet" />




Footnotes


What does rotch mean? Is it even a word? No, it is not. It’s a surname for a few Americans: a meteorologist, an architect, a tennis player, two politicians and a pediatrician.↩︎

Garrick Aden-Buie

Remembering what I did this week with LLMs

The object impermanence of work

What did I even do this week?

Hiring my executive robot assistant

System prompt

Work context

The user prompt

The final result

Professional Work

Personal Work

Using local LLMS

Summary of Recent Work

Professional Activities:

Open Source Contributions:

Final script

Footnotes

It’s time to add bslib to your shinyapp snippet

Before (default snippet)

After (with bslib)

Bonus: A VSCode snippet

Add last rendered or modified time to Quarto

A blog is a perpetual side-project machine

Hello, now

Using now

Formatting the time output

Have a good day, meow!

Automated Image Overflow Detection for xaringan or remark Slides

Plain text slides

PDF slides and the case of the missing images

More slides, more problems

Example slides

Automate it!

With chromote

Match window size to slide size

Read and Visualize your Twitter Archive

Reading your Twitter archive

Get your Twitter data archive

Setup

Read the manifest

countdown v0.4.0 – Now on CRAN!

What is countdown?

Installing countdown

A brief history of countdown

rstudio::conf(2019)

It becomes an R package

But that JavaScript code…

coundown v0.4.0

The Old JavaScript

All New JavaScript

New buttons and keyboard interactions

Shiny!

Improved countdown App

New Options

Thank you!

Footnotes

Process Profile Pictures with magick

rstudio::conf(2022) runs on R

Send me your profile picture, please

Collect the Profile Pictures

Magick with R

Finding Faces

Resize

Find Resized Faces

Cropping

Put it all together

Footnotes

xaringanExtra v0.6.0 — Now on CRAN!

What is xaringanExtra?

Installing xaringanExtra

Wait, it’s on CRAN now?

A new extra: banner!

Saving Daylight Time?

Are we saving daylight in Atlanta, GA?

Around the World

Across the US

Inspiration

Where are you?

Sunrise and Sunset Times

Tidy Sun Times