R

Twice a year, the tidyverse team sets a week aside for “spring cleaning,” bringing all of our R packages up to snuff with the most current tooling and standardizing various bits of our development process. Some of these updates can happen by calling a single function, while others are much more involved. One of those more involved updates is updating erroring code, transitioning away from base R (e.g.  stop()), rlang (e.g.  rlang::abort()), glue, and homegrown combinations of them. cli’s new syntax is easier to work with as a developer and more visually pleasing as a user.

In some cases, transitioning is almost as simple as Finding + Replacing rlang::abort() to cli::cli_abort():

# before: rlang::abort("`save_pred` can only be used if the initial results saved predictions.") # after: cli::cli_abort("{.arg save_pred} can only be used if the initial results saved predictions.")

In others, there’s a mess of ad-hoc pluralization, paste0()s, glue interpolations, and other assorted nonsense to sort through:

Thus was born clipal1, a (now-superseded) R package that allows users to select erroring code, press a keyboard shortcut, wait a moment, and watch the updated code be inlined in to the selection.

clipal was a huge boost for us in the most recent spring cleaning. Depending on the code being updated, these erroring calls used to take 30 seconds to a few minutes. With clipal, though, the model could usually get the updated code 80% or 90% of the way there in a couple seconds. Up to this point, irritated by autocomplete and frustrated by the friction of copying and pasting code and typing out the same bits of context into chats again and again, I had been relatively skeptical that LLMs could make me more productive. After using clipal for a week, though, I began to understand how seamlessly LLMs could automate the cumbersome and uninteresting parts of my work.

clipal itself is now superseded by pal, a more general solution to the problem that clipal solved. I’ve also written two additional packages like pal that solve two other classes of pal-like problems using similar tools, ensure and gander. In this post, I’ll write a bit about how I’ve used a pair of tools in three experiments that have made me much more productive as an R developer

After using clipal during our spring cleaning, I approached another spring cleaning task for the week: updating testing code. testthat 3.0.0 was released in 2020, bringing with it numerous changes that were both huge quality of life improvements for package developers and also highly breaking changes. While some of the task of converting legacy unit testing code to testthat 3e is relatively straightforward, other components can be quite tedious. Could I do the same thing for updating to testthat 3e that I did for transitioning to cli? I sloppily threw together a sister package to clipal that would convert tests for errors to snapshot tests, disentangle nested expectations, and transition from deprecated functions like ⁠expect_known_*(). ⁠(If you’re interested, the current prompt for that functionality is here.) That sister package was also a huge boost for me, but the package reused as-is almost every piece of code from clipal other than the prompt. Thus, I realized that the proper solution would provide all of this scaffolding to attach a prompt to a keyboard shortcut, but allow for an arbitrary set of prompts to help automate these wonky, cumbersome tasks.

The next week, pal was born. The pal package ships with three prompts centered on package development: the cli pal and testthat pal mentioned previously, as well as the roxygen pal, which drafts minimal roxygen documentation based on a function definition. Here’s what pal’s interface looks like now:

ensure

While deciding on the initial set of prompts that pal would include, I really wanted to include some sort of “write unit tests for this function” pal. To really address this problem, though, requires violating two of pal’s core assumptions:

All of the context that you need is in the selection and the prompt. In the case of writing unit tests, it’s actually pretty important to have other pieces of context. If a package provides some object type potato, in order to write tests for some function that takes potato as input, it’s likely very important to know how potatoes are created and the kinds of properties they have. pal’s sister package for writing unit tests, ensure, can thus “see” the rest of the file that you’re working on, as well as context from neighboring files like other .R source files, the corresponding test file, and package vignettes, to learn about how to interface with the function arguments being tested.

The LLM’s response can prefix, replace, or suffix the active selection in the same file. In the case of writing unit tests for R, the place that tests actually ought to go is in a corresponding test file in tests/testthat/. Via the RStudio API, ensure can open up the corresponding test file and write to it rather than the source file where it was triggered from.3

gander

Examples @examples provides executable R code showing how to use the function in practice. This is a very important part of the documentation because many people look at the examples before reading anything. Example code must work without errors as it is run automatically as part of R CMD check. For the purpose of illustration, it’s often useful to include code that causes an error. You can do this by wrapping the code in try() or using \dontrun{} to exclude from the executed example code. For finer control, you can use @examplesIf: #' @examplesIf interactive() #' browseURL("https://roxygen2.r-lib.org")

Instead of including examples directly in the documentation, you can put them in separate files and use @example path/relative/to/package/root to insert them into the documentation. All functions must have examples for initial CRAN submission.

It is recommended to use the same structure in your tests/testthat/ directory as your R/ directory, i.e., a test file for every .R file.

formattable Another nice table-making package is formattable. The cute heatmap-style colour formatting and the easy-to-use formatter functions make formattable very appealing. color_tile() fills the cells with a colour gradient corresponding to the values color_bar() adds a colour bar to each cell, where the length is proportional to the value The true_false_formatter() defined below demonstrates how to define your own formatting function, in this case formatting TRUE, FALSE and NA as green, red and black.

If you want the features of both DT and formattable, you can combine them by converting the formattable() output to as.datatable(), and much of the formattable features will be preserved.

However, one problem I had was that when using DT::datatable, missing values (NA) are left blank in the display (which I prefer), but in the converted from formattable() version, NA’s are printed. Also, color_bar columns seem to be converted to character, which can no longer be sorted numerically.

reactable Next I tried reactable, a package based on the React Table library.

Columns are customised via the columns argument, which takes a named list of column definitions defined using colDef(). These include format definitions created using colFormat.

In the end, I used DT::datatable() in my Shiny app, because I found it the easiest, fastest, and most comprehensive. I’ve been able to achieve most of the features I wanted using just DT.

Heatmap-like fill effect:

apply the formatStyle() function to the output of datatable() to set the backgroundColor for selected columns:

Abbreviate long cells

Sometimes some cells have a large amount of text that would mess up the table layout if I showed it all. In these cases, I like to abbreviate long values and show the full text in a tooltip. To do this, you can use JavaScript to format the column to show a substring with “…” and the full string in a tooltip (<span title="Long string">Substring...</span) when values are longer than N characters (in this case 10 characters). You can do this using columnDefs and pass JavaScript with the JS() function:

Really plain table Sometimes I don’t need any of the faff. Here’s how to get rid of it all:

headerCallbackRemoveHeaderFooter <- c( "function(thead, data, start, end, display){", " $('th', thead).css('display', 'none');", "}" )

datatable( my_pic_villagers, options = list( dom = "t", ordering = FALSE, paging = FALSE, searching = FALSE, headerCallback = JS(headerCallbackRemoveHeaderFooter) ), selection = 'none', callback = JS( "$('table.dataTable.no-footer').css('border-bottom', 'none');" ), class = 'row-border', escape = FALSE, rownames = FALSE, filter = "none", width = 500 )

Compared to event-based programming, reactivity allows Shiny to do the minimum amount of work when input(s) change, and allows humans to more easily reason about complex MVC logic.

An attractive default look based on Bootstrap which can also be easily customized with the bslib package or avoided entirely with more direct R bindings to HTML/CSS/JavaScript.

Tools for improving and monitoring performance, including native support for async programming, caching, load testing, and more.

observeEvent() is used to perform an action in response to an event eventReactive() is used to create a calculated value that only updates in response to an event

observe() and reactive() functions automatically trigger on whatever they access observeEvent() and eventReactive() functions need to be explicitly told what triggers them

And where does isolate fit in all this? isolate() is used to stop a reaction observeEvent() is used to perform an action in response to an event eventReactive() is used to create a calculated value that only updates in response to an event

Create .Renviron file Within the get_data.R script of my repository, I extract my EIA API key from my R environment so that I can connect to the EIA API and pull the data needed for my project. In order for this to occur during my workflow, I need to create an .Renviron file within my virtual environment and store the key within that environment. - name: Create and populate .Renviron file run: | echo EIA_API_KEY="$EIA_API_KEY" >> ~/.Renviron shell: bash

Using Crosstalk to Add User-Interactivity

The goal is to link the reactable table I created to a plotly chart and provide additional filter options that control both the table and the chart.

An important note: in order to use crosstalk, you must create a shared dataset and call that dataset within both plotly and reactable. Otherwise, your dataset will not communicate and filter with eachother. The code to do this is SharedData$new(dataset).

If you expand the code below, you’ll see that the code to build a table in reactable is quite extensive. I will not go into the details in this post, but do recommend a couple great tutorials that I used to create the interactive table such as this tutorial from Greg Lin, and this from Tom Mock which really helped me understand how to use CSS and Google fonts to enhance the visual appeal of the table (see the “Additional CSS Used for Table” section below for more info).

If you have ever built something in Shiny before, you’ll notice that the crosstalk filters are very similar. You can add a filter to any existing column in the dataset. As you can see in the code below, I used a mixture of filter_checkbox and filter_select depending on how many unique options were available in the column you’re filtering. My rule of thumb is if there are more than five options to choose from it’s probably better to put them into a list in filter_select like I did with the Division filtering as to not take up too much space on the page.

For the layout of the data visualization, I used bscols to place the crosstalk filters side-by-side with the interactive plotly chart. I then placed the reactable table underneath and added a legend to the table using tags from the htmltools package. The final result is shown below. Feel free to click around and the filters and you will notice that both the plot and the table will filter accordingly. Another option is to drag and click on the plot and you will see the table underneath mimic the teams shown.

Snapshot testing gets difficult when there is more than one variant of the same result. The reason why snapshot testing might be discouraging is due to the fact that snapshots will most likely fail due to environment settings. If one person runs the tests on a Mac and another on a Linux machine, the snapshots of rendered images will almost certainly be different. Comparing these snapshots will result in a failed test even though the code is correct. Add CI to the mix, and you have a hot mess.

The easiest solution is to introduce variants. Variants are versions of snapshots which were created on different environments. In {testthat} variants are stored in separate directories. You can pass a name of the variant to the variant argument of testthat::test_snapshot. If you have a Linux, set variant = "linux", if you have a Mac, set variant = "mac".

Use snapshots generated on CI as the source of truth. Don’t check in snapshots generated on your machine. Generate them on CI and download them to your machine instead.

Step 1: Archive snapshots on CI Add this step to you CI testing workflow to allow downloading generated snapshots.

- name: Archive test snapshots if: always() uses: actions/upload-artifact@v3 with: name: test-snapshots path: | tests/testthat/_snaps/**/**/*

Step 2: Detect the environment to create variants We can create a make_variant function to detect the version of the platform, as well as if we are running on CI. This way even if we use the same OS on CI and locally, we can still differentiate between snapshots generated on CI and locally.

#' tests/testthat/setup.R is_ci <- function() { isTRUE(as.logical(Sys.getenv("CI"))) } make_variant <- function(platform = shinytest2::platform_variant()) { ci <- if (is_ci()) "ci" else NULL paste(c(ci, platform), collapse = "-") } # In tests: testthat::expect_snapshot(..., variant = make_variant())

Step 3: Ignore your local snapshots Don’t check in snapshots generated on your machine. Add them to .gitignore instead. Copy tests/testthat/_snaps/linux-4.4 This way we can still generate snapshots locally to get fast feedback, but we’ll only keep a single source of truth checked in the repository. Since you don’t track changes in local snapshots, you need to regenerate them before you start making changes to see if they change. It adds some complexity to the process, but it allows to keep the number of shared snapshots in the version control minimal. Alternatively, you can keep local snapshots, but when doing code review, focus only on the ones generated on CI.

Step 4: Automate downloading snapshots from CI To update snapshots generated on CI in Github, we need to: Go to Actions. Find our workflow run. Download the test-snapshots artifact. Unpack and overwrite the local snapshots. testthat::snapshot_review() to review the changes. Commit and push the changes. This is a lot of steps. We can automate the most laborious ones with Github API.

The .download_ci_snaps function will: Get the list of artifacts in the repository identified by repo and owner. It’ll search workflows generated from the branch we’re currently on. It will download the latest artifact with the provided name (in our case its “test-snapshots”) in the repository Unzip them and overwrite the local copy of snapshots.