Lab Notebook

“Data rectangling”: the process of turning highly nested data structures (e.g. JSON, XML) into a tabular format. Data rectangling is a brilliant turn of phrase coined by Jenny Bryan (UBC, RStudio) and leader in the #rstats community. Recording or slides of Jenny’s talk on the subject give a much better intro to the idea and working with this in R, particularly through the purrr package. As nice as purrr is for the task, I’ve recently found that the jqr package from Scott Chamberlain and co can be a much easier way to go about rectangling your JSON.

Using JSON Queries (JQ) Pure JSON vita <- readr::read_file("../../static/js/vita.json") jq(vita, '."@reverse".author[] | { year: .dateCreated, author: .author[] | [.givenName, .familyName] | join(" ") }') %>% combine() %>% fromJSON() year author 2017-12-09 Wayne M. Getz 2017-12-09 Charles R. Marshall 2017-12-09 Colin J. Carlson 2017-12-09 Luca Giuggioli 2017-12-09 Sadie J. Ryan 2017-12-09 Stephanie S. Romañach 2017-12-09 Carl Boettiger 2017-12-09 Samuel D. Chamberlain 2017-12-09 Laurel Larsen 2017-12-09 Paolo D’Odorico 2017-12-09 David O’Sullivan 2017-03-15 Stephanie E.

In general I’m a big believer in the R package mechanism as a convenient way to package and distribute a reproducible compendium of code, data, notebooks and manuscripts. However, it’s often overkill, particularly since passing basic continuous integration checks (e.g. via devtools::use_travis()) means running a relatively strict R CMD check that will worry about things such as documentation norms and details that aren’t always appropriate. What I most want is an easy way to confirm that any .

Parsing very nested JSON documents can be a pain. Here are some notes on co-opting the strategy of “Framing” used in JSON-LD. (Note that unlike the basic operations of compaction and expansion, the JSON-LD framing algorithm actually is essentially independent of the @context and any linked data concepts. Here’s a toy example of some nested JSON. Very nested structures are usually the source of issues for me, even with purrr, because often I want to pull data found at various different levels of nesting into a single row for the data.

R 3.4.0 and devel tags are now available in the Rocker versioned stack, and latest has been updated to 3.4.0. This means you can now run something like: docker run -d -p 8787:8787 rocker/tidyverse:devel to test out a recent version1 of R devel in RStudio with the tidyverse packages already installed. Note that the packages are not themselves devel versions (e.g. from GitHub) but are instead the latest versions from CRAN, just as in the latest tag.

After OpenWetWare (2010), Wordpress (2010 - 2012) and Jekyll (2012 - 2017), I move platforms once again: now to Hugo. Why the move? As usual, this site remains a way for me to explore new technologies, but the cutting edge is by no means the allure which leads me to migrate. Rather, I find myself ever trying to move towards simpler platforms, what’s easier to maintain, free to host, easier to replicate, and what makes best use of the most widely available and well-maintained existing software and avoids unncessary custom hacks.