02-AREAS

02-AREAS

1893 bookmarks
Newest
UNCHARTED DATA: Automating Workflows with GitHub Actions
UNCHARTED DATA: Automating Workflows with GitHub Actions
How to automate data collection and app deployment with GitHub Actions.
Create .Renviron file Within the get_data.R script of my repository, I extract my EIA API key from my R environment so that I can connect to the EIA API and pull the data needed for my project. In order for this to occur during my workflow, I need to create an .Renviron file within my virtual environment and store the key within that environment. - name: Create and populate .Renviron file run: | echo EIA_API_KEY="$EIA_API_KEY" >> ~/.Renviron shell: bash
·uncharteddata.netlify.app·
UNCHARTED DATA: Automating Workflows with GitHub Actions
UNCHARTED DATA: Using Crosstalk to Add User-Interactivity
UNCHARTED DATA: Using Crosstalk to Add User-Interactivity
Linking an interactive plot and table together with the crosstalk package.
Using Crosstalk to Add User-Interactivity
The goal is to link the reactable table I created to a plotly chart and provide additional filter options that control both the table and the chart.
An important note: in order to use crosstalk, you must create a shared dataset and call that dataset within both plotly and reactable. Otherwise, your dataset will not communicate and filter with eachother. The code to do this is SharedData$new(dataset).
If you expand the code below, you’ll see that the code to build a table in reactable is quite extensive. I will not go into the details in this post, but do recommend a couple great tutorials that I used to create the interactive table such as this tutorial from Greg Lin, and this from Tom Mock which really helped me understand how to use CSS and Google fonts to enhance the visual appeal of the table (see the “Additional CSS Used for Table” section below for more info).
If you have ever built something in Shiny before, you’ll notice that the crosstalk filters are very similar. You can add a filter to any existing column in the dataset. As you can see in the code below, I used a mixture of filter_checkbox and filter_select depending on how many unique options were available in the column you’re filtering. My rule of thumb is if there are more than five options to choose from it’s probably better to put them into a list in filter_select like I did with the Division filtering as to not take up too much space on the page.
For the layout of the data visualization, I used bscols to place the crosstalk filters side-by-side with the interactive plotly chart. I then placed the reactable table underneath and added a legend to the table using tags from the htmltools package. The final result is shown below. Feel free to click around and the filters and you will notice that both the plot and the table will filter accordingly. Another option is to drag and click on the plot and you will see the table underneath mimic the teams shown.
·uncharteddata.netlify.app·
UNCHARTED DATA: Using Crosstalk to Add User-Interactivity
Design Patterns in R
Design Patterns in R
Build robust and maintainable software with object-oriented design patterns in R. Design patterns abstract and present in neat, well-defined components and interfaces the experience of many software designers and architects over many years of solving similar problems. These are solutions that have withstood the test of time with respect to re-usability, flexibility, and maintainability. R6P provides abstract base classes with examples for a few known design patterns. The patterns were selected by their applicability to analytic projects in R. Using these patterns in R projects have proven effective in dealing with the complexity that data-driven applications possess.
·tidylab.github.io·
Design Patterns in R
The most efficient way to manage snapshot tests in R.
The most efficient way to manage snapshot tests in R.
Use CI and Github API
Snapshot testing gets difficult when there is more than one variant of the same result. The reason why snapshot testing might be discouraging is due to the fact that snapshots will most likely fail due to environment settings. If one person runs the tests on a Mac and another on a Linux machine, the snapshots of rendered images will almost certainly be different. Comparing these snapshots will result in a failed test even though the code is correct. Add CI to the mix, and you have a hot mess.
The easiest solution is to introduce variants. Variants are versions of snapshots which were created on different environments. In {testthat} variants are stored in separate directories. You can pass a name of the variant to the variant argument of testthat::test_snapshot. If you have a Linux, set variant = "linux", if you have a Mac, set variant = "mac".
Use snapshots generated on CI as the source of truth. Don’t check in snapshots generated on your machine. Generate them on CI and download them to your machine instead.
Step 1: Archive snapshots on CI Add this step to you CI testing workflow to allow downloading generated snapshots.
- name: Archive test snapshots if: always() uses: actions/upload-artifact@v3 with: name: test-snapshots path: | tests/testthat/_snaps/**/**/*
Step 2: Detect the environment to create variants We can create a make_variant function to detect the version of the platform, as well as if we are running on CI. This way even if we use the same OS on CI and locally, we can still differentiate between snapshots generated on CI and locally.
#' tests/testthat/setup.R is_ci <- function() { isTRUE(as.logical(Sys.getenv("CI"))) } make_variant <- function(platform = shinytest2::platform_variant()) { ci <- if (is_ci()) "ci" else NULL paste(c(ci, platform), collapse = "-") } # In tests: testthat::expect_snapshot(..., variant = make_variant())
Step 3: Ignore your local snapshots Don’t check in snapshots generated on your machine. Add them to .gitignore instead. Copy tests/testthat/_snaps/linux-4.4 This way we can still generate snapshots locally to get fast feedback, but we’ll only keep a single source of truth checked in the repository. Since you don’t track changes in local snapshots, you need to regenerate them before you start making changes to see if they change. It adds some complexity to the process, but it allows to keep the number of shared snapshots in the version control minimal. Alternatively, you can keep local snapshots, but when doing code review, focus only on the ones generated on CI.
Step 4: Automate downloading snapshots from CI To update snapshots generated on CI in Github, we need to: Go to Actions. Find our workflow run. Download the test-snapshots artifact. Unpack and overwrite the local snapshots. testthat::snapshot_review() to review the changes. Commit and push the changes. This is a lot of steps. We can automate the most laborious ones with Github API.
The .download_ci_snaps function will: Get the list of artifacts in the repository identified by repo and owner. It’ll search workflows generated from the branch we’re currently on. It will download the latest artifact with the provided name (in our case its “test-snapshots”) in the repository Unzip them and overwrite the local copy of snapshots.
·jakubsob.github.io·
The most efficient way to manage snapshot tests in R.
Prompt Storm - A Powerful Easy to use Artificial Intelligence Prompt Engineering Chrome Software Extension for ChatGPT, Google's Gemini, and Anthropic's Claude.
Prompt Storm - A Powerful Easy to use Artificial Intelligence Prompt Engineering Chrome Software Extension for ChatGPT, Google's Gemini, and Anthropic's Claude.
Prompt Storm - A Powerful Easy to use AI Prompt Engineering Chrome Extension for ChatGPT, Google's Gemini, and Anthropic's Claude. With just a few clicks you can get the answers you're looking for, create amazing writing, marketing and social media strategies, save time and boost your productivity.
·promptstorm.app·
Prompt Storm - A Powerful Easy to use Artificial Intelligence Prompt Engineering Chrome Software Extension for ChatGPT, Google's Gemini, and Anthropic's Claude.
HTTP resources and specifications - HTTP | MDN
HTTP resources and specifications - HTTP | MDN
HTTP was first specified in the early 1990s. Designed with extensibility in mind, it has seen numerous additions over the years; this lead to its specification being scattered through numerous specification documents (in the midst of experimental abandoned extensions). This page lists relevant resources about HTTP.
·developer.mozilla.org·
HTTP resources and specifications - HTTP | MDN
Using References
Using References
For API designers and writers wishing formalize their API in an OpenAPI Description document.
Using References in OpenAPI Descriptions OpenAPI Referencing is a powerful tool. It allows managing document size and complexity, and allows re-using shared components and avoiding copy-paste or change management errors. However, the history of referencing and the "$ref" keyword is complex, leading to slightly different behavior depending on the version of the OpenAPI Specification (OAS) that you are using, and on where in your OpenAPI Description (OAD) the reference occurs. There are also other "$ref"-like keywords ("operationRef", "mapping") and behaviors (referencing by component name or operation ID) that are related but somewhat different. Referencing can also be challenging to use due to incomplete and inconsistent support across different tools, some of which require references to be pre-processed before they can read the OAD. The resources in this section explain how to use referencing, and what to look for when assessing the referencing support in your OpenAPI tools.
The Future of References There are plans to reduce the complexity around references in future OpenAPI Specifications. The Moonwalk project is considering a different approach that imports complete documents, associates them with namespaces, and only supports referencing by component name (not "$ref"). A small example can be seen in the Moonwalk deployments proposal, and there are discussions around an initial draft proposal for imports and a few ideas on how to manage interactions with JSON Schema referencing. The proposed Workflows Specification is already using a "sourceDescription" field that is not unlike the Moonwalk proposal.
·learn.openapis.org·
Using References
Best Practices
Best Practices
For API designers and writers wishing formalize their API in an OpenAPI Description document.
Keep a Single Source of Truth Regardless of your design approach (design-first or code-first) always keep a single source of truth, i.e., information should not be duplicated in different places. It is really the same concept used in programming, where repeated code should be moved to a common function.
Otherwise, eventually one of the places will be updated while the other won’t, leading to headaches… in the best of cases. For instance, it is also commonplace to use code annotations to generate an OpenAPI description and then commit the latter to source control while the former still lingers in the code. As a result, newcomers to the project will not know which one is actually in use and mistakes will be made. Alternatively, you can use a Continuous Integration test to ensure that the two sources stay consistent.
Add OpenAPI Descriptions to Source Control OpenAPI Descriptions are not just a documentation artifact: they are first-class source files which can drive a great number of automated processes, including boilerplate generation, unit testing and documentation rendering. As such, OADs should be committed to source control, and, in fact, they should be among the first files to be committed. From there, they should also participate in Continuous Integration processes.
Make the OpenAPI Descriptions Available to the Users Beautifully-rendered documentation can be very useful for the users of an API, but sometimes they might want to access the source OAD. For instance, to use tools to generate client code for them, or to build automated bindings for some language. Therefore, making the OAD available to the users is an added bonus for them. The documents that make up the OAD can even be made available through the same API to allow runtime discovery.
There is Seldom Need to Write OpenAPI Descriptions by Hand Since OADs are plain text documents, in an easy-to-read format (be it JSON or YAML), API designers are usually tempted to write them by hand. While there is nothing stopping you from doing this, and, in fact, hand-written API descriptions are usually the most terse and efficient, approaching any big project by such method is highly impractical. Instead, you should try the other existing creation methods and choose the one that better suits you and your team (No YAML or JSON knowledge needed!):
OpenAPI Editors: Be it text editors or GUI editors they usually take care of repetitive tasks, allow you to keep a library of reusable components and provide real-time preview of the generated documentation.
Domain-Specific Languages: As its name indicates, DSL’s are API description languages tailored to specific development fields. A tool is then used to produce the OpenAPI Description. A new language has to be learned, but, in return, extremely concise descriptions can be achieved.
Code Annotations: Most programming languages allow you to annotate the code, be it with specific syntax or with general code comments. These annotations, for example, can be used to extend a method signature with information regarding the API endpoint and HTTP method that lead to it. A tool can then parse the code annotations and generate OADs automatically. This method fits very nicely with the code-first approach, so keep in mind the first advice given at the top of this page when using it (Use a Design-First Approach)…
A Mix of All the Above: It’s perfectly possible to create the bulk of an OpenAPI Description using an editor or DSL and then hand-tune the resulting file. Just be aware of the second advice above (Keep a Single Source of Truth): Once you modify a file it becomes the source of truth and the previous one should be discarded (maybe keep it as backup, but out of the sight and reach of children and newcomers to the project).
Describing Large APIs
Do not repeat yourself (The DRY principle). If the same piece of YAML or JSON appears more than once in the document, it’s time to move it to the components section and reference it from other places using $ref (See Reusing Descriptions. Not only will the resulting document be smaller but it will also be much easier to maintain). Components can be referenced from other documents, so you can even reuse them across different API descriptions!
Split the description into several documents: Smaller files are easier to navigate, but too many of them are equally taxing. The key lies somewhere in the middle. A good rule of thumb is to use the natural hierarchy present in URLs to build your directory structure. For example, put all routes starting with /users (like /users and /users/{id}) in the same file (think of it as a “sub-API”). Bear in mind that some tools might have issues with large files, whereas some other tools might not handle too many files gracefully. The solution will have to take your toolkit into account.
Use tags to keep things organized: Tags have not been described in the Specification chapter, but they can help you arrange your operations and find them faster. A tag is simply a piece of metadata (a unique name and an optional description) that you can attach to operations. Tools, specially GUI editors, can then sort all your API’s operation by their tags to help you keep them organized.
Links to External Best Practices There’s quite a bit of literature about how to organize your API more efficiently. Make sure you check out how other people solved the same issues you are facing now! For example: The API Stylebook contains internal API Design Guidelines shared with the community by some well known companies and government agencies.
Best Practices This page contains general pieces of advice which do not strictly belong to the Specification Explained chapter because they are not directly tied to the OpenAPI Specification (OAS). However, they greatly simplify creating and maintaining OpenAPI Descriptions (OADs), so they are worth keeping in mind.
Use a Design-First Approach Traditionally, two main approaches exist when creating OADs: Code-first and Design-first. In the Code-first approach, the API is first implemented in code, and then its description is created from it, using code comments, code annotations or simply written from scratch. This approach does not require developers to learn another language so it is usually regarded as the easiest one. Conversely, in Design-first, the API description is written first and then the code follows. The first obvious advantages are that the code already has a skeleton upon which to build, and that some tools can provide boilerplate code automatically. There have been a number of heated debates over the relative merits of these two approaches but, in the opinion of the OpenAPI Initiative (OAI), the importance of using Design-first cannot be stressed strongly enough.
The reason is simple: The number of APIs that can be created in code is far superior to what can be described in OpenAPI. To emphasize: OpenAPI is not capable of describing every possible HTTP API, it has limitations. Therefore, unless these descriptive limitations are perfectly known and taken into account when coding the API, they will rear their ugly head later on when trying to create an OpenAPI description for it. At that point, the right fix will be to change the code so that it uses an API which can be actually described with OpenAPI (or switch to Design-first altogether). Sometimes, however, since it is late in the process, it will be preferred to twist the API description so that it matches more or less the actual API. It goes without saying that this leads to unintuitive and incomplete descriptions, that will rarely scale in the future. Finally, there exist a number of validation tools that can verify that the implemented code adheres to the OpenAPI description. Running these tools as part of a Continuous Integration process allows changing the OpenAPI Description with peace of mind, since deviations in the code behavior will be promptly detected. Bottom line: OpenAPI opens the door to a wealth of automated tools. Make sure you use them!
·learn.openapis.org·
Best Practices
Fast JSON, NDJSON and GeoJSON Parser and Generator
Fast JSON, NDJSON and GeoJSON Parser and Generator
A fast JSON parser, generator and validator which converts JSON, NDJSON (Newline Delimited JSON) and GeoJSON (Geographic JSON) data to/from R objects. The standard R data types are supported (e.g. logical, numeric, integer) with configurable handling of NULL and NA values. Data frames, atomic vectors and lists are all supported as data containers translated to/from JSON. GeoJSON data is read in as simple features objects. This implementation wraps the yyjson C library which is available from .
·coolbutuseless.github.io·
Fast JSON, NDJSON and GeoJSON Parser and Generator