NSSP-ESSENCE RStudio API Guide
Introduction
RStudio lets you access ESSENCE data more securely, create your own R Markdown reports and Shiny applications, and do exploratory analyses not possible within ESSENCE. In this R Markdown document, we will provide an overview of the ESSENCE application programming interfaces (APIs) and explain how to access them through RStudio. We will also list the APIs available in ESSENCE and show basic examples of R code and packages so that you can start using RStudio.
An API is a structured and consistent way for one machine to exchange information with another machine. ESSENCE has APIs that allow you to programmatically access and further manipulate your data from outside the system. You may select the CSV format to export data, and in some instances, JSON is also supported. More information about the APIs can be found in ESSENCE under “More,” then “User Guide,” and then API Documentation. You may write API URL syntax on your own after reading the documentation, or you can let ESSENCE create the API URL by clicking the “API URL” button on an ESSENCE page after completing a query.
10 ESSENCE APIs are highlighted in this guide:
Time series data table,
Time series PNG image,
Table builder results,
Data details (line level),
Summary stats on the number of unique facilities or regions in your query results,
Alert list detection tables,
Facility-level data quality metrics,
ESSENCE query fields,
CCDD category table,
Time series data table with stratified, historical alerts
RStudio: Securely Saving AMC Credentials
The first step is to create an R script or R Markdown file and load the necessary packages. By default, some packages might already be in your system library. But if not, you can install them yourself by clicking on the packages tab in the bottom-right quadrant of the RStudio interface and selecting the Install tab. Here are the package names and configuration statement for getting started:
library(Rnssp)
library(tidyverse)
library(lubridate)
library(httr)
library(jsonlite)To extract data from ESSENCE, you start by passing authentication information from your RStudio session to ESSENCE so that the system knows what data you are allowed to see. It is bad practice to explicitly include your username and password in your code. This guide will promote the secure methodology supplied by the Rnssp R package for authentication of AMC credentials.
As of April 2024, Rnssp version 0.3.0 is installed system-wide on the BioSense Platform instance of Posit Workbench. If you are using a desktop (local) instance of RStudio, you can install Rnssp from GitHub by running devtools::install_github(“cdcgov/Rnssp”) in your console. The Rnssp GitHub repository can be accessed at https://github.com/CDCgov/Rnssp, with additional documentation and vignettes located at https://cdcgov.github.io/Rnssp/. Rnssp provides functionality to securely save credentials and interact with ESSENCE APIs. When you run the following code, a pop-up will appear in RStudio for you to enter your AMC username and password. This will create a user profile object of the class NSSPCredentials, designed with the R6 object system that integrates classic object-oriented programming concepts into R.
Note
The example below should be used within an R script. It is possible to save your myProfile object to your home directory as either an .rda file or .rds file. If you save your myProfile object, you can easily load it into your global environment for authenticating your credentials. The create_profile() step only needs to be run once every 90 days when you update your AMC password. Additionally, adding the create_profile() line to a code chunk within an R Markdown or Quarto Markdown document will prevent the file from rendering. Please only use this function in an R script or in your R console.
The “~” in the file path specifies that the profile object will be saved in your home directory, which only you will have access to. Please do not save your myProfile object to Posit Workbench shared projects that other users have access to. It is imperative to always save the myProfile object to your home directory with the use of “~” in the file path.
myProfile <- create_profile()
# rda option
save(myProfile, file = "~/myProfile.rda")
# rds option
saveRDS(myProfile, "~/myProfile.rds")An alternative method for saving your credential profile object to your home directory is to use the Create User Profile (GUI) RStudio addin. To use this option, follow the steps below:
- Select “Create User Profile (GUI)” from the Addins dropdown menu located at the top of the RStudio session. This will open a small window (GUI) asking for credential information.
- Enter your AMC username and password.
- Check the box for “Save Profile to Home Directory”.
- Choose the desired file format (
.rdaor.rds) - Click the blue “Done” button.
For screen captures of this process and more details on the Rnssp RStudio Addins, please see: Using Rnssp RStudio Addins.
To load your myProfile object into your global environment, use the load() function for an .rda file and the readRDS() function for an .rds file. To run ESSENCE API pulls within an R Markdown or Quarto Markdown file, simply add the appropriate load/read option to an initial code chunk in the file.
# .rda option
load("~/myProfile.rda")
# .rds option
myProfile <- readRDS("~/myProfile.rds")Note that your username and password are fully encrypted in your myProfile object and are not visible when viewing or inspecting the object:
myProfile<NSSPCredentials>
Public:
clone: function (deep = FALSE)
get_api_data: function (url, fromCSV = FALSE, ...)
get_api_graph: function (url, file_ext = ".png")
get_api_response: function (url)
initialize: function (username = NULL, password = NULL)
Private:
..__: NSSPContainer, R6
..password: NSSPContainer, R6
..username: NSSPContainer, R6The myProfile object comes with the $get_api_response(), $get_api_data(), and $get_api_graph() methods with various parameters to pull ESSENCE data:
$get_api_response(): Retrieves requested information specified in the API URL from ESSENCE, which is useful for inspecting the HTTR status code for debugging purposes$get_api_data(): Extracts the content (data) from the API response and parses the data into an R data frame$get_api_graph(): Retrieves an ESSENCE time series graph and saves it as a PNG file within a temporary directory
Alternatively, the get_api_response(), get_api_data(), and get_api_graph() functions serve as wrappers to their respective methods. As of 2024, NSSP recommends using the wrapper functions out of convenience. With previous versions of Rnssp, users would traditionally use the approach shown below to pull ESSENCE data for a given API URL, url:
api_data <- myProfile$get_api_data(url)Users can now simply use the wrapper function, get_api_data(), without having to type myProfile$:
api_data <- get_api_data(url)The get_api_data() function accepts the following arguments with defaults specified below:
api_data <- get_api_data(url, fromCSV = FALSE, profile = myProfile, ...)url: A character of API URLfromCSV: A logical defining whether data are returned in CSV or JSON formatprofile: An object of classCredentials, with default beingmyProfile...: Additional arguments and CSV parsing parameters to be passed to theread_csv()function from thereadrpackage whenfromCSV = TRUE
Keep in mind that the API for time series data and value set lists will always import in JSON format. Only the table builder and data details APIs provide the option to choose between CSV and JSON. The examples included in this guide will expand upon this further.
In each example in the subsequent sections in this guide, you will see a common pattern emerge: First, the URL must be defined as an object in your RStudio session; then the API response needs to be retrieved and content extracted to an R data frame using the get_api_data() function before further analysis. Retrieval of the API response is done implicitly when calling get_api_data(). Explicit retrieval of the API response using get_api_response() is only recommended for troubleshooting purposes. When using the approaches outlined in this guide, R Markdown and Quarto provides an easily reproducible workflow for integrating report text with code that reads in data, manipulates data as needed, and produces analyses and visualizations in such a way that can be handed off to ESSENCE users access without documenting manual actions (point/click, etc). Please keep in mind that all examples in this guide use the limited details data sources in NSSP-ESSENCE. Apart from more availability of stratification options and data details variables, the process for each example will be identical for API pulls run again the full details data sources.
Time Series Data Table
The first example demonstrates how to pull a time series data table from ESSENCE into RStudio. The API response retrieval and content extraction steps are included to provide a view of what get_api_data() implicitly implements for time series API pulls. In the code snippet below, the first object created is the URL for the ESSENCE API endpoint of interest, which for this example is a time series data table containing daily national counts for the injury syndrome. This URL was generated within the ESSENCE user interface by running the query and copy-pasting the URL from the “API URLs” option under “Query Options”.
The second object, api_response, authenticates your credentials so that the ESSENCE system knows you are permitted to access these data and then retrieves the requested data if authentication is successful. An HTTP status code of 200 indicates success. If the request is not successful, you might see one of the following status codes:
- 401: Unauthorized - usually due to an expired AMC password or a typo in the username and/or password supplied to the
myProfileobject - 400: Bad Request - usually due to syntax errors (see example bullets below) in the API URL when edited by the user outside of ESSENCE
- Supplying invalid date syntax for start and end date
- Misspelling of a query field or parameter
The next two objects process JSON-formatted data into an R data frame. The fromJSON() function converts these data to a list of length 2, where the second element of the list is a data frame that can be extracted with the pluck() function from the purrr library (included in tidyverse). Lastly, the glimpse() function from dplyr provides a quick sense of every variable in the data frame.
url <- "https://essence.syndromicsurveillance.org/nssp_essence/api/timeSeries?endDate=6Apr2024&medicalGrouping=injury&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=7Jan2024&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries"
api_response <- get_api_response(url)
api_response_json <- httr::content(api_response, as = "text")
api_data <- jsonlite::fromJSON(api_response_json) %>%
pluck("timeSeriesData")
glimpse(api_data)- 1
-
Define an object,
url, with the ESSENCE-generated API URL - 2
-
Retrieve the API response using
get_api_response() - 3
-
Use the
content()function from thehttrpackage to extract JSON-formatted data as text - 4
-
Use the
fromJSON()function from thejsonlitepackage to convert the list element containing JSON-formatted data to an R data frame - 5
-
Use the
pluck()function from thepurrrpackage to extract the dataframe from the list returned in step 4 - 6
-
The
glimpse()function can be used to preview your variables, data, and variable types
Rows: 91
Columns: 11
$ date <chr> "2024-01-07", "2024-01-08", "2024-01-09", "2024-01-10", "202…
$ count <dbl> 41140, 49197, 46950, 48600, 48862, 47019, 42640, 41200, 4467…
$ expected <dbl> 41948.48, 48878.92, 47576.36, 46861.23, 46688.66, 47014.06, …
$ levels <dbl> 0.711, 0.415, 0.648, 0.121, 0.072, 0.499, 0.713, 0.662, 0.03…
$ colorID <int> 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 2, 1, 1, 0, 1, 3, 2, 1, 1, 1, …
$ color <chr> "grey", "blue", "blue", "blue", "blue", "blue", "grey", "gre…
$ altText <chr> "Data: Date: 07Jan24, Level: 0.711, Count: 41140, Expected: …
$ details <chr> "/nssp_essence/api/dataDetails?percentParam=noPercent&dataso…
$ graphType <chr> "absolute", "absolute", "absolute", "absolute", "absolute", …
$ lineLabel <chr> "Data", "Data", "Data", "Data", "Data", "Data", "Data", "Dat…
$ title <chr> "Daily Data Counts", "Daily Data Counts", "Daily Data Counts…As mentioned previously, these data can be simply pulled with two lines of code by using the get_api_data() function, which implicitly implements the steps outlined above for a time series data pull. Occasionally there are scenarios in which retrieving the API response is beneficial—for example, when the API response needs to be inspected if data are not pulling as expected. In general, the direct get_api_data() approach below is recommended. The remaining examples in this guide do not use get_api_response().
api_data <- get_api_data(url) %>%
pluck("timeSeriesData")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.Another alternative to pull data using the ESSENCE APIs is to use the get_essence_data() Rnssp function. This convenience function was designed to auto-detect the data type (i.e., time series, table builder, data details, or alert list) and data format (CSV vs. JSON) by inspecting the API URL argument, url. Additionally, the function allows users to supply start_date and end_date arguments to modify the dates in the API URL. The get_essence_data() function accepts the following arguments with defaults specified below:
api_data <- get_essence_data(url, start_date = NULL, end_date = NULL, profile = myProfile, ...)url: A character of ESSENCE API URLstart_date: A date object or a character string in date format (e.g., “2019-08-01”)end_date: A date object or a character string in date format (e.g., “2020-08-01”)profile: An object of class NSSPCredentials...: Further arguments and CSV parsing parameters to be passed to theread_csv()function from thereadrpackage
To modify the start and end dates in the time series API URL example from above, the function is used as follows:
api_data <- get_essence_data(url, start_date = "2024-01-01", end_date = "2024-11-09")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.ESSENCE Time Series Graph
This example shows how to retrieve the ESSENCE graph instead of the underlying data for the graph, as shown previously. The API URL for the time series graph is copy-pasted from the “Graph (PNG)” option from “API URLs” under “Query Options” (see Figure 1). Users are able to specify custom graph titles or axes labels by adding parameters to the ESSENCE-generated URL: “&graphTitle=Injury%20Syndrome&xAxisLabel=Date&yAxisLabel=Count”.
url <- "https://essence.syndromicsurveillance.org/nssp_essence/api/timeSeries/graph?endDate=6Apr2024&medicalGrouping=injury&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=7Jan2024&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries&graphTitle=National%20-%20Injury%20Syndrome%20Daily%20Counts&xAxisLabel=Date&yAxisLabel=Count"
api_ts_graph <- get_api_graph(url)knitr::include_graphics(api_ts_graph$graph, rel_path = FALSE)
Table Builder Results
With the table builder API, ESSENCE does the heavy lifting by summarizing your query and presenting results in tabular format where users can define rows, nested rows, and column variables for output. If the query is supported in the ESSENCE query manager, users should be able to use table builder to summarize the output, which is usually more efficient than manipulating large amounts of line-level data outside of the system. In this example, the CDC Opioid Overdose v3 Chief Complaint Discharge Diagnosis (CCDD) category is used to create a table of counts per month by U.S. Department of Health & Human Services (HHS) region and NCHS age group. Visits are limited to emergency department visits by selecting Has been Emergency = “Yes”. Output formats for table builder results are CSV and JSON. The CSV option will pull in data that matches the tabular format seen in the ESSENCE interface, whereas the JSON option will pull in data that is transformed to a long, pivoted format. The latter is recommended to eliminate extra steps to transform data into a long format that is compatible with functions/libraries based on tidyverse principles.
CSV Example
When the CSV option is used, the fromCSV argument of get_api_data() must be set to TRUE as the default argument value is FALSE and assumes the JSON option. In this example, month (time resolution) and HHS region were selected as row fields and NCHS age group as a column field in the ESSENCE table builder user interface. Once selections for row and column fields are made, ESSENCE will auto-populate the corresponding API URL under Query Options.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/tableBuilder/csv?endDate=30Apr2024&percentParam=noPercent&datasource=va_hospdreg&startDate=1Jan2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TableBuilder&ccddCategory=cdc%20opioid%20overdose%20v3&geographySystem=hospitaldhhsregion&detector=nodetectordetector&timeResolution=monthly&hasBeenE=1&rowFields=timeResolution&rowFields=geographyhospitaldhhsregion&columnField=ageNCHS"
api_data <- get_api_data(url, fromCSV = TRUE)ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.Rows: 44 Columns: 13
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): timeResolution, geographyhospitaldhhsregion
dbl (11): 00-10, 11-14, 15-24, 25-34, 35-44, 45-54, 55-64, 65-74, 75-84, 85+...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.glimpse(api_data)Rows: 44
Columns: 13
$ timeResolution <chr> "2024-01", "2024-01", "2024-01", "2024-01"…
$ geographyhospitaldhhsregion <chr> "OTHER_REGION", "Region 1", "Region 10", "…
$ `00-10` <dbl> 0, 5, 9, 7, 14, 31, 12, 7, 3, 3, 7, 0, 2, …
$ `11-14` <dbl> 0, 1, 2, 1, 9, 3, 4, 7, 1, 1, 5, 0, 2, 3, …
$ `15-24` <dbl> 0, 55, 149, 75, 105, 395, 170, 152, 53, 63…
$ `25-34` <dbl> 0, 278, 500, 349, 340, 1120, 584, 328, 63,…
$ `35-44` <dbl> 0, 357, 438, 377, 407, 1154, 621, 302, 49,…
$ `45-54` <dbl> 0, 256, 230, 292, 288, 795, 448, 181, 32, …
$ `55-64` <dbl> 0, 225, 254, 356, 345, 755, 468, 153, 47, …
$ `65-74` <dbl> 0, 106, 209, 154, 151, 566, 324, 113, 35, …
$ `75-84` <dbl> 0, 23, 76, 28, 29, 289, 65, 46, 16, 30, 65…
$ `85+` <dbl> 0, 10, 16, 16, 11, 92, 24, 16, 5, 8, 33, 0…
$ Unknown <dbl> 0, 25, 35, 16, 40, 58, 20, 7, 0, 2, 12, 0,…As NCHS age group was chosen as the column field, there are columns in the data representing counts for each age group. These column names therefore start with a numeric character and contain dashes to represent the range of ages covered in each group, which means that the age group columns will need to be surrounded in backticks anytime they are referenced in code. In R it is best practice (and in some instances, required) to adhere to a set of standards for column names within a dataframe. Column names should:
- Ideally only use lower case letters
- Not contain spaces, only underscores
- Never start with a number
- Not contain special characters such as !, $, @, %, #, *
To improve efficiency of coding and references to variable names, the best solution in this scenario would be to pivot the data to a longer format where the information in the age group columns is represented by 2 new columns: 1) a column age_nchs containing character values representing the age ranges (i.e., 00-10, 11-14, etc.); and 2) a visit_count column containing the monthly counts for each age group. Additionally, use of the clean_names() function from the janitor package is an efficient option to ensure other column names are standardized. In this example, the clean_names() function would convert the column name timeResolution to time_resolution. As a last step, the dataframe is filtered to remove “OTHER_REGION” since the data source is based on facility location which is always known.
df_long <- api_data %>%
pivot_longer(-c(timeResolution, geographyhospitaldhhsregion), names_to = "age_nchs", values_to = "visit_count") %>%
janitor::clean_names() %>%
filter(geographyhospitaldhhsregion != "OTHER_REGION")
df_long# A tibble: 440 × 4
time_resolution geographyhospitaldhhsregion age_nchs visit_count
<chr> <chr> <chr> <dbl>
1 2024-01 Region 1 00-10 5
2 2024-01 Region 1 11-14 1
3 2024-01 Region 1 15-24 55
4 2024-01 Region 1 25-34 278
5 2024-01 Region 1 35-44 357
6 2024-01 Region 1 45-54 256
7 2024-01 Region 1 55-64 225
8 2024-01 Region 1 65-74 106
9 2024-01 Region 1 75-84 23
10 2024-01 Region 1 85+ 10
# ℹ 430 more rowsJSON Example
If the JSON option is used, there is no need to include fromCSV = FALSE in the get_api_data() call since FALSE is the default value. This example will demonstrate that the pivoting transformation used in the CSV example is not necessary and that the data are already imported in a long format. Note that the only difference between the JSON and CSV API URLs is the “/csv” string is no longer present after the “api/tableBuilder” string towards the beginning of the URL.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/tableBuilder?endDate=30Apr2024&percentParam=noPercent&datasource=va_hospdreg&startDate=1Jan2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TableBuilder&ccddCategory=cdc%20opioid%20overdose%20v3&geographySystem=hospitaldhhsregion&detector=nodetectordetector&timeResolution=monthly&hasBeenE=1&rowFields=timeResolution&rowFields=geographyhospitaldhhsregion&columnField=ageNCHS"
api_data <- get_api_data(url) %>%
janitor::clean_names() %>%
filter(geographyhospitaldhhsregion != "OTHER_REGION") ℹ Success: (200) OKglimpse(api_data)Rows: 440
Columns: 4
$ time_resolution <chr> "2024-01", "2024-01", "2024-01", "2024-01"…
$ geographyhospitaldhhsregion <chr> "Region 1", "Region 1", "Region 1", "Regio…
$ age_nchs <chr> "00-10", "11-14", "15-24", "25-34", "35-44…
$ count <dbl> 5, 1, 55, 278, 357, 256, 225, 106, 23, 10,…
Important
Please keep in mind that the table builder in the ESSENCE user interface is limited to creating tables of up to 30,000 cells. A table this large or smaller should suffice for most query needs. The table builder API however, does not impose a limit on the output table size.
Data Details (line-level limited details)
Sometimes users just need line-level data, and this example describes how to extract those data from ESSENCE. This method will create a very large data set quickly. As with any query with potential to create large data sets, first run the query as a time series to see the number of associated records. If the time series reveals that millions of records will be pulled, users should consider multiple calls of smaller time ranges before combining the separate dataframes to create a final data set. Additionally, the data details APIs give users the option to only pull a subset of variables in data details and the option to pull raw or reference values. Data details API pulls with reference values may take longer than raw values as ESSENCE must create the reference values first. Examples of raw vs. reference values include Region IV (raw) vs. Region 4 (reference) and 65-1000 (raw) vs. 65+ (reference). Output formats for data details are CSV and JSON. The CSV option is typically more efficient for larger data details pulls.
Example 1 - CSV
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/dataDetails/csv?endDate=29May2024&geography=region%20i&percentParam=noPercent&datasource=va_hospdreg&startDate=28May2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=DataDetails&medicalGrouping=injury&geographySystem=hospitaldhhsregion&detector=nodetectordetector&timeResolution=daily&hasBeenE=1"
api_data <- get_api_data(url, fromCSV = TRUE)ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.Rows: 6121 Columns: 30
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (21): Date, Category_flat, SubCategory_flat, Patient_Class, HospitalDHH...
dbl (8): HasBeenE, HasBeenI, HasBeenO, HasBeenAdmitted, DDAvailable, DDInf...
dttm (1): FirstDateTimeAdded
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.glimpse(api_data)Rows: 6,121
Columns: 30
$ Date <chr> "05/28/2024", "05/28/2024", "05/28/2024", "…
$ Category_flat <chr> ";Injury;", ";Injury;", ";Injury;", ";Injur…
$ SubCategory_flat <chr> ";Fall;", ";CutOrPierce;", ";Fall;", ";Fall…
$ Patient_Class <chr> "E", "E", "E", "E", "E", "E", "E", "E", "E"…
$ HospitalDHHSRegion <chr> "Region I", "Region I", "Region I", "Region…
$ dhhsregion <chr> "Region I", "Region I", "Region I", "Region…
$ AgeGroup <chr> "18-44", "65-1000", "65-1000", "05-17", "65…
$ Sex <chr> "F", "M", "F", "F", "M", "M", "F", "M", "F"…
$ CCDDCategory_flat <chr> "none", "none", ";CDC AFM Broad v1-Limit to…
$ Race_flat <chr> ";2106-3;", ";2106-3;", ";2106-3;", ";2106-…
$ Ethnicity_flat <chr> ";2186-5;", ";2186-5;", ";2186-5;", ";2186-…
$ DispositionCategory <chr> "DISCHARGED", "DISCHARGED", "DISCHARGED", "…
$ ICD_CCSR_Desc_Flat <chr> ";Exposure; encounters; screening or contac…
$ ICD_CCSR_Flat <chr> ";FAC016;INJ012;INJ017;INJ027;", ";FAC016;I…
$ ICD_Chapter_Desc_Flat <chr> ";Injury; poisoning and certain other conse…
$ ICD_Chapter_Flat <chr> ";19;21;", ";19;21;", ";18;", ";19;", ";1;1…
$ ICD_Section_Desc_Flat <chr> ";Injuries to the head (S00-S09);Injuries t…
$ ICD_Section_Flat <chr> ";S00-S09;S50-S59;Z20-Z29;", ";S60-S69;Z20-…
$ AdmissionTypeCategory <chr> "NR", "NR", "NR", "NR", "NR", "NR", "NR", "…
$ HasBeenE <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ HasBeenI <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0…
$ HasBeenO <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ HasBeenAdmitted <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0…
$ DDAvailable <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ DDInformative <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ CCAvailable <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ CCInformative <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ FirstDateTimeAdded <dttm> 2024-05-28 11:21:36, 2024-05-28 16:02:12, …
$ CRace_CEth_Combined_Broad <chr> "White and non-Hispanic", "White and non-Hi…
$ CRace_CEth_Combined_Narrow <chr> "White and non-Hispanic", "White and non-Hi…Example 2 - JSON
As with the time series example, the JSON API option for data details will return a list of length 1. The data details dataframe can be extracted from this list by using pluck("dataDetails"):
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/dataDetails?endDate=29May2024&geography=region%20i&percentParam=noPercent&datasource=va_hospdreg&startDate=28May2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=DataDetails&medicalGrouping=injury&geographySystem=hospitaldhhsregion&detector=nodetectordetector&timeResolution=daily&hasBeenE=1"
api_data <- get_api_data(url) %>%
pluck("dataDetails")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.glimpse(api_data)Rows: 6,121
Columns: 30
$ Date <chr> "05/28/2024", "05/28/2024", "05/28/2024", "…
$ Category_flat <chr> ";Injury;", ";Injury;", ";Injury;", ";Injur…
$ SubCategory_flat <chr> ";Fall;", ";CutOrPierce;", ";Fall;", ";Fall…
$ Patient_Class <chr> "E", "E", "E", "E", "E", "E", "E", "E", "E"…
$ HospitalDHHSRegion <chr> "Region I", "Region I", "Region I", "Region…
$ dhhsregion <chr> "Region I", "Region I", "Region I", "Region…
$ AgeGroup <chr> "18-44", "65-1000", "65-1000", "05-17", "65…
$ Sex <chr> "F", "M", "F", "F", "M", "M", "F", "M", "F"…
$ CCDDCategory_flat <chr> "none", "none", ";CDC AFM Broad v1-Limit to…
$ Race_flat <chr> ";2106-3;", ";2106-3;", ";2106-3;", ";2106-…
$ Ethnicity_flat <chr> ";2186-5;", ";2186-5;", ";2186-5;", ";2186-…
$ DispositionCategory <chr> "DISCHARGED", "DISCHARGED", "DISCHARGED", "…
$ ICD_CCSR_Desc_Flat <chr> ";Exposure, encounters, screening or contac…
$ ICD_CCSR_Flat <chr> ";FAC016;INJ012;INJ017;INJ027;", ";FAC016;I…
$ ICD_Chapter_Desc_Flat <chr> ";Injury, poisoning and certain other conse…
$ ICD_Chapter_Flat <chr> ";19;21;", ";19;21;", ";18;", ";19;", ";1;1…
$ ICD_Section_Desc_Flat <chr> ";Injuries to the head (S00-S09);Injuries t…
$ ICD_Section_Flat <chr> ";S00-S09;S50-S59;Z20-Z29;", ";S60-S69;Z20-…
$ AdmissionTypeCategory <chr> "NR", "NR", "NR", "NR", "NR", "NR", "NR", "…
$ HasBeenE <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1"…
$ HasBeenI <chr> "0", "0", "0", "0", "0", "0", "0", "0", "0"…
$ HasBeenO <chr> "0", "0", "0", "0", "0", "0", "0", "0", "0"…
$ HasBeenAdmitted <chr> "0", "0", "0", "0", "0", "0", "0", "0", "0"…
$ DDAvailable <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1"…
$ DDInformative <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1"…
$ CCAvailable <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1"…
$ CCInformative <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1"…
$ FirstDateTimeAdded <chr> "2024-05-28 11:21:36.033", "2024-05-28 16:0…
$ CRace_CEth_Combined_Broad <chr> "White and non-Hispanic", "White and non-Hi…
$ CRace_CEth_Combined_Narrow <chr> "White and non-Hispanic", "White and non-Hi…Example 3 - Limiting to specified fields
A great option to reduce the size of a data details data set is to specify a minimal subset of variables needed for an analysis. This can be done by adding additional parameters to the end of the URL that indicate the columns of interest. For example, if a user only wanted to include date, HHS region, age group, and patient sex, “&field=Date&field=HospitalDHHSRegion&field=AgeGroup&field=Sex” would be manually added to the end of the API URL.
Important
Limiting to a subset of desired fields is much more efficient when pulling line-level data from the full details data sources, which includes 216 total variables as of May 2024. Documentation on proper naming convention for full data details fields can be found in the ESSENCE API Query Parameters tab of the NSSP Data Dictionary.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/dataDetails/csv?endDate=29May2024&geography=region%20i&percentParam=noPercent&datasource=va_hospdreg&startDate=28May2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=DataDetails&medicalGrouping=injury&geographySystem=hospitaldhhsregion&detector=nodetectordetector&timeResolution=daily&hasBeenE=1&field=Date&field=HospitalDHHSRegion&field=AgeGroup&field=Sex"
api_data <- get_api_data(url, fromCSV = TRUE)ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.Rows: 6121 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (4): Date, HospitalDHHSRegion, AgeGroup, Sex
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.glimpse(api_data)Rows: 6,121
Columns: 4
$ Date <chr> "05/28/2024", "05/29/2024", "05/28/2024", "05/28/20…
$ HospitalDHHSRegion <chr> "Region I", "Region I", "Region I", "Region I", "Re…
$ AgeGroup <chr> "18-44", "00-04", "05-17", "18-44", "65-1000", "18-…
$ Sex <chr> "M", "M", "M", "F", "F", "F", "F", "M", "F", "F", "…Example 4 - Time chunked data pulls
As mentioned previously, when users pull line-level data details over a longer range of time (i.e., months or a year), it is ideal to time chunk the API data pulls to reduce burden on the ESSENCE system. Please keep in mind that heavy queries can affect system performance for all ESSENCE users. The following example is meant for demonstrative purposes and only pulls data for 4 days. This can be used as a template to pull data for longer periods of time. The steps used within this workflow are explained upon hovering over the numbered steps on the right side of the code chunk below.
api_data <- data.frame(date = seq.Date(as.Date("2024-05-26"), as.Date("2024-05-29"), by = "1 day")) %>%
mutate(
date = format(date, "%d%b%Y"),
url = paste0("https://essence.syndromicsurveillance.org/nssp_essence/api/dataDetails/csv?medicalGrouping=injury&geography=region%20i&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=dataDetails&startDate=", date, "&endDate=", date)
) %>%
mutate(pull = map(.x = url, ~get_api_data(., fromCSV = TRUE))) %>%
unnest(pull) %>%
select(-c(date, url))- 1
- Define a sequence of dates to be queried and stored as column in a dataframe where each row corresponds to a single date in the sequence
- 2
- Format dates to the formatting required in ESSENCE API URLs (i.e., 26May2024)
- 3
- Create a column of ESSENCE API URLs where the formatted dates are pasted in for start and end date values
- 4
-
Use the
map()function frompurrrto applyget_api_data()to each day’s API URL and create a column of dataframes with each containing data details for the corresponding day - 5
-
Use the
unnest()function to combine all dataframes into one data set - 6
-
Deselect the
dateandurlcolumns that were used to build the API URLs since they are no longer needed
Summary Stats
The Summary Stats ESSENCE API will count the number of unique facilities and ESSENCE regions in a query by the time resolution selected by the user (daily, weekly, monthly, quarterly, or yearly). In the ESSENCE user interface, Summary Stats will also display the top 50 words in the chief complaints of records identified by the query as well as the mean, median, standard deviation, minimum, and maximum number of words in the chief complaint. The Summary Stats API will only return information on the number of unique facilities and regions.
url <- "https://essence.syndromicsurveillance.org/nssp_essence/api/summaryData?endDate=2Sep2024&medicalGrouping=injury&geography=region%20i&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hosp&detector=probrepswitch&startDate=1Sep2024&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries"
api_data <- get_api_data(url) %>%
pluck("summaryData")ℹ Success: (200) OKglimpse(api_data)Rows: 2
Columns: 5
$ date <chr> "01Sep24", "02Sep24"
$ HospitalState <dbl> 6, 6
$ State <dbl> 27, 25
$ Region <dbl> 142, 134
$ Hospital <dbl> 236, 229Temporal Alert List Detection Tables
The Temporal Alert List API gives you programmatic access to the temporal alert list tables in the ESSENCE user interface. Temporal alert tables based on syndrome definitions are available at the region (or, “county”) and county Federal Information Processing Standards (FIPS) code approximation levels. A facility-level alert list is only available for ESSENCE chief complaint syndromes.
The results in these tables are updated a few times daily. Alert lists based on patient location include:
- Patient Region - Syndrome: “Region/Syndrome”
- Patient FIPS Approximation - Syndrome: “FIPS/Syndrome”
- Patient Region - Subsyndrome: “Region/SubSyndrome”
- Patient FIPS Approximation - Subsyndrome: “FIPS/SubSyndrome”
- Patient Region - CCDD: “Region/CCDD”
- Patient FIPS Approximation - CCDD: “FIPS/CCDD”
Alert lists based on facility location include:
- Facility Region - Syndrome: “Hospital/Syndrome”
- Facility FIPS Approximation - Syndrome: “FacilityFIPS/Syndrome”
- Facility FIPS Approximation - Subsyndrome: “FacilityFIPS/SubSyndrome”
- Facility FIPS Approximation - CCDD: “FacilityFIPS/CCDD”
Additionally, term-based alerts became available in ESSENCE2 as of July 2021. Examples of pulling word alerts using the Alert List API are presented in the New Functionality Section. The majority of the alert lists only maintain alerts from the last 30 days. If historical data exceeding this time range are pulled, an empty (NULL) data frame will be returned. Note: All users have access to patient and hospital region-level alerts. You will only be able to drill down to live-level data details for alerts occurring in regions from which you have data access. For all alert list examples included in this guide, previews of output data are not included due to the level of granularity and to address privacy concerns.
Patient Region
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/alerts/regionSyndromeAlerts?end_date=8Sep2024&start_date=7Sep2024"
api_data <- get_api_data(url) %>%
pluck("regionSyndromeAlerts")ℹ Success: (200) OKHospital Region
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/alerts/hospitalSyndromeAlerts?end_date=8Sep2024&start_date=7Sep2024"
api_data <- get_api_data(url) %>%
pluck("hospitalSyndromeAlerts")ℹ Success: (200) OKSpatial Alert Detection Lists
In addition to the temporal alert lists, the Alert List tab in ESSENCE contains 3 alert lists based on the ESSENCE spatial cluster detection algorithm, which is a Java-based implementation of the spatial algorithm implemented in the SaTScan software. Technical details of the ESSENCE spatial cluster determination algorithm can be found in the ESSENCE Users Guide under the “More” tab. The 3 spatial alert lists are based on patient location (patient zip code) and are available for: 1) Syndromes; 2) Subsyndromes; and 3) CCDD categories. Please keep in mind that as of 2024, this algorithm is purely spatial and not spatiotemporal. In this guide, an example API URL for Syndromes is provided.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/alerts/spatialSyndromeAlerts?end_date=8Sep2024&start_date=7Sep2024"
api_data <- get_api_data(url) %>%
pluck("spatialSyndromeAlerts") ℹ Success: (200) OKESSENCE myAlerts
“myAlerts” is an ESSENCE function that syndromic surveillance practitioners can use to add additional filtering and parameter constraints on alerting criteria. These alerts are produced by an automated process which can run on a routine basis defined by the user. myAlerts includes two primary functions:
- Enables creation of custom alerts for syndromes, subsyndromes, CCDD categories, or free-text queries by various stratifications or threshold criteria.
- Enables creation of Records of Interest. The purpose of Records of Interest is to return encounter details that match you query criteria (e.g., the word or code for anthrax)
To set up a “myAlert” in the ESSENCE UI, please refer to the ESSENCE Users Guide.
To pull all “myAlerts” defined by or shared with your account within the ESSENCE user interface, use the following standard API URL:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/myAlerts/alerts"
api_data <- get_api_data(url) %>%
pluck("alerts")The following API URL can be used to pull records of interest from “myAlerts”:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/myAlerts/recordsOfInterest"
api_data <- get_api_data(url) %>%
pluck("recordsOfInterest")New Functionality
Limiting Time Series Fields (Added 2024)
Often times, users will need to run time series queries that import a large amount of data. This may cause issues on the R-side if the total number of characters in the returned JSON object exceeds the R character maximum (2^31-1 bytes). One possible solution is to limit the fields included in the time series data as you can do with data details API pulls. Below is a simple example to illustrate the full set of fields included in a time series query:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/timeSeries?endDate=10Sep2024&percentParam=noPercent&datasource=va_hospdreg&startDate=12Jun2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TimeSeries&ccddCategory=all%20traffic%20related%20v2&patientLocDHHSRegion=region%20ii&geographySystem=hospitaldhhsregion&detector=probrepswitch&timeResolution=daily&hasBeenE=1"
api_data <- get_api_data(url) %>%
pluck("timeSeriesData")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.glimpse(api_data)Rows: 91
Columns: 11
$ date <chr> "2024-06-12", "2024-06-13", "2024-06-14", "2024-06-15", "202…
$ count <dbl> 776, 873, 885, 859, 837, 962, 972, 802, 971, 848, 824, 714, …
$ expected <dbl> 849.846, 850.407, 848.500, 843.893, 847.679, 850.036, 852.67…
$ levels <dbl> 0.870, 0.415, 0.286, 0.370, 0.496, 0.052, 0.018, 0.246, 0.05…
$ colorID <int> 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ color <chr> "blue", "blue", "blue", "blue", "blue", "blue", "yellow", "b…
$ altText <chr> "Data: Date: 12Jun24, Level: 0.87, Count: 776, Expected: 849…
$ details <chr> "/nssp_essence/api/dataDetails?percentParam=noPercent&dataso…
$ graphType <chr> "absolute", "absolute", "absolute", "absolute", "absolute", …
$ lineLabel <chr> "Data", "Data", "Data", "Data", "Data", "Data", "Data", "Dat…
$ title <chr> "Daily Data Counts", "Daily Data Counts", "Daily Data Counts…Typically, users will only be interested in using fields like date, count, and anomaly detection information fields. Columns like “altText” and “details” often contain long character strings that can contributed to this problem. To remove them, use a similiar approach as data details:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/timeSeries?endDate=10Sep2024&percentParam=noPercent&datasource=va_hospdreg&startDate=12Jun2024&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TimeSeries&ccddCategory=all%20traffic%20related%20v2&patientLocDHHSRegion=region%20ii&geographySystem=hospitaldhhsregion&detector=probrepswitch&timeResolution=daily&hasBeenE=1&field=date&field=count&field=expected&field=levels&field=color"
api_data <- get_api_data(url) %>%
pluck("timeSeriesData")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.glimpse(api_data)Rows: 91
Columns: 5
$ date <chr> "2024-06-12", "2024-06-13", "2024-06-14", "2024-06-15", "2024…
$ count <dbl> 776, 873, 885, 859, 837, 962, 972, 802, 971, 848, 824, 714, 9…
$ expected <dbl> 849.846, 850.407, 848.500, 843.893, 847.679, 850.036, 852.679…
$ levels <dbl> 0.870, 0.415, 0.286, 0.370, 0.496, 0.052, 0.018, 0.246, 0.052…
$ color <chr> "blue", "blue", "blue", "blue", "blue", "blue", "yellow", "bl…ESSENCE Data Quality Tables (Added 2022)
Tables containing facility-level data quality metrics are located under the Data Quality tab in the ESSENCE user interface. The coefficient of variation (CoV) measures the overall volatility of weekly facility volume over a specified time period (i.e., 1 year to date, 2 years to date, etc.). The CoV is calculated by dividing the standard deviation by the mean of weekly encounters over this period and multiplying by 100 to report as a percentage. Small CoV values (i.e., < 35%) represent facilities with consistent overall volume and reporting, while high CoV values represent facilities with volatile weekly volume that are potentially attributable to facility onboarding during the specified time period or data drop outs due to data quality or reporting issues. As of December 2022, NSSP-ESSENCE includes the following data quality CoV filters:
- Data Quality CoV Current Year to Date
- Data Quality CoV Last Year to Date
- Data Quality CoV Last 2 Years to Date
- Data Quality CoV Last 3 Years to Date
- Data Quality CoV Last 4 Years to Date
- Data Quality CoV Last 5 Years to Date
There are 2 options for each of these filters: 1) general CoV filters for all encounters; 2) HasBeenE filters for emergency department encounters. Users can select operators associated with the defined CoV threshold within the ESSENCE user interface. These operators include “Equal”, “Does Not Equal”, “Less Than”, “Less Than or Equal”, “Greater Than”, “Greater Than or Equal”, “Between”, or “Is Null”. The “Less Than” or “Less Than or Equal” operators are most commonly used with CoV thresholds between 30% and 40%.
In 2022, a new API was added to allow ESSENCE users to pull CoV statistics for facilities reporting data to NSSP-ESSENCE. This API can be used to pull information for all facilities or limited to facilities within a specific site. The first API pull below pulls data for all facilities, while the second example limits to facilities in Florida by specifying the appropriate siteid value at the end of the API URL.
# All NSSP-ESSENCE facilities
api_url1 <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/dataQuality/COVWklyAvgData"
cov_data1 <- get_api_data(api_url1)
# Limited to facilities in a specific site
api_url2 <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/dataQuality/COVWklyAvgData?siteid=884"
cov_data2 <- get_api_data(api_url2)
dim(cov_data1)[1] 8720 172These data include 172 total fields. In addition to CoV information, the data include facility-level metrics for informativeness of discharge diagnosis and chief complaint, total volume, and date of last update. The HTML table below displays all variables and their category for use.
Pulling Query Field Values (Added November 2021)
Users can now pull values for available query fields in ESSENCE2 via the APIs. As an example, you may want to pull the list of all current CCDD categories, syndromes, and subsyndromes. The Syndrome Subsyndrome CCDD Combined Category query field in ESSENCE2 provides a complete list of all existing syndrome definitions. The following API URL can be used as a template to pull values from other query fields. Metadata will be pulled in JSON format.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/QueryWizard/ParamMetaData.json?action=getParameterMetaData&datasourceId=va_er¶mName=combinedCategory&parentParamValue=&parentParamName=&filterParamValList="
syndromes <- get_api_data(url) %>%
pluck("valueDisplayFields")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.head(syndromes, 10) valueField
1 a_ccdd_acute hepatitis b v1
2 a_ccdd_air quality-related respiratory illness v1
3 a_ccdd_all traffic related v1
4 a_ccdd_all traffic related v2
5 a_ccdd_cdc acute flaccid myelitis dd v1
6 a_ccdd_cdc acute hepatitis c v1
7 a_ccdd_cdc afm broad v1-limit to pediatric
8 a_ccdd_cdc afm narrow v1-limit to pediatric
9 a_ccdd_cdc alcohol v1
10 a_ccdd_cdc all drug overdose v3 parsed
displayField
1 CCDD Acute Hepatitis B v1
2 CCDD Air Quality-related Respiratory Illness v1
3 CCDD All Traffic Related v1
4 CCDD All Traffic Related v2
5 CCDD CDC Acute Flaccid Myelitis DD v1
6 CCDD CDC Acute Hepatitis C v1
7 CCDD CDC AFM Broad v1-Limit to Pediatric
8 CCDD CDC AFM Narrow v1-Limit to Pediatric
9 CCDD CDC Alcohol v1
10 CCDD CDC All Drug Overdose v3 ParsedPulling CCDD Category Table (Added 2022)
Alternatively, you may want to pull in an up-to-date list of all existing ESSENCE CCDD categories and their underlying queries. This is useful for providing additional context in category-specific reports or summaries.
url <- "https://essence.syndromicsurveillance.org/nssp_essence/servlet/SyndromeDefinitionsServlet_CCDD?action=getCCDDTerms"
ccdd_queries <- get_api_data(url) %>%
pluck("categories")ℹ Success: (200) OKccdd_queries %>%
filter(category == "Heat Related Illness v1") updateInfo termId groupName dateCreated notes lastUpdate
1 No Updates 30 Uncategorized 2019-04-11 No Notes 2019-09-09
description
1 Description to be added
definition
1 Heat Related Illness Inclusions (CC and DD): \n(,^[;/ ]992^, OR,(,^[;/ ]E900^,ANDNOT,^[;/ ]E900.1^,ANDNOT,^[;/ ]E9001^,), OR,^[;/ ]T67^, OR,^[;/ ]X30^,ANDNOT,^[;/ ]W92^,), OR,(,(,^HEAT ^, OR,^HEATCRAMP^, OR,^HEATEX^, OR,^HEATST^, OR,^HYPERTHERM^, OR,^OVERHEAT^, OR,^OVER HEAT^, OR,^SUNSTR^, OR,^SUN STR^, OR,^TO HOT^, OR,^TOO HOT^, OR,(,(,^HEET^, OR,^HOT^,),AND,(,^EXCESSIVE^, OR,^EXHAUST^, OR,^EXPOS^, OR,^FATIGUE^, OR,^CRAMP^, OR,^STRESS^, OR,^IN CAR^, OR,^OUTSIDE^, OR,^PROSTRATION^,),),),ANDNOT,(,^ALLERG^, OR,^FEELING HOT^, OR,^FEELS HOT^, OR,^FELT HOT^, OR,(,^HOT^,AND,^SENSATION^,), OR,^HEAT SENSATION^, OR,^ INFLAM^ , OR,(,^PAIN^, AND,(,^LIMB^, OR,^ARM^, OR,^SHOULDER^, OR,^ELBOW^, OR,^WRIST^, OR,^HAND^, OR,^LEG^, OR,^HIP^, OR,^GROIN^, OR,^THIGH^, OR,^KNEE^, OR,^ANKLE^, OR,^FOOT^, OR,^FEET^, OR,^BACK^, OR,^NECK^, OR,^FLANK^, OR,^RED^, OR,^JAW^, OR,^MOUTH^, OR,^TEETH^, OR,^TOOTH^,),), OR,^RADIAT^, OR,^REDNESS^, OR,^SWELL^, OR,^SWOLLEN^, OR,^SURG^, OR,^POST OP^, OR,^IBUPROFEN^, OR,^IBUPROPHEN^, OR,^ALIEVE^, OR,^MOTRIN^, OR,^TYLENOL^, OR,^INJUR^, OR,^TRAUMA^, OR,(,^HEAT^,AND,(,^ICE^, OR,^APPLIED^, OR,^TRIED^, OR,^USED^, OR,^USING^, OR,^COLD^, OR,^RASH^,),), OR,^HEAT PACK^, OR,^HEATING PAD^, OR,^LUMBAGO^, OR,^RELIEF^, OR,^RESOLVE^, OR,^RELIEVE^, OR,^RELEIVE^, OR,^DENTAL^, OR,(,^HOT^,AND,(,^COLD^, OR,^COFF^, OR,^SHOWER^,),), OR,(,^ORAL^,AND,^SURG^,), OR,(,^SENSITIV^,AND,(,^HEAT^, OR,^HOT^,),), OR,^HOT DOG^, OR,^HOT GREASE^, OR,^HOT EPPERS^, OR,^HOT TEA^, OR,^HEAT ACHE^, OR,^HEATACHE^, OR,^HEAT ATTACK^, OR,^HEAT BEAT^, OR,^HEATBEAT^, OR,^HEAT BURN^, OR,^HEATBURN^, OR,^HEAT FLUTTER^, OR,^HEAT RACING^, OR,^HEAT RATE^, OR,^HEATRATE^, OR,^HEATLH^, OR,^HEATH^, OR,^HEATTH^, OR,^HITTING HEAT^, OR,^PALPITATION^, OR,^CHEAT^, OR,^HEATER^, OR,^HEATHER^, OR,^HEATING^, OR,^HOTEL^, OR,^LITHOTR^, OR,^METHOTR^, OR,^PHOTO^, OR,^PSYCHOTIC^, OR,^SHEATH^, OR,^SHEET^, OR,^SHOT^, OR,^SUNDAY^, OR,^THEAT^, OR,^WHEAT^, OR,^ACCIDENT^, OR,^ALCOHOL^, OR,(,^BURN^,AND,^MOUTH^,), OR,^DISTRESS^, OR,^FEVER^, OR,^GETS HOT^, OR,^HEAT FLASH^, OR,^HOT FLASH^, OR,^HIVES^, OR,^HOT TUB^, OR,^NO HEAT^, OR,^OVEN^, OR,^SUICID^,),)\n<p>\nHeat Related Illness Exclusions(Cheif Complaint): \n^,ANDNOT,^X30^
isAdmin category fieldsSearched
1 TRUE Heat Related Illness v1 UndefinedWord Alerts (Added July 2021)
Term-based alerts are available in ESSENCE2. This algorithm was developed to identify unusual distributions of chief complaint terms of interest without dependence on syndrome definitions. Technical details of the word alert algorithm can be found at https://essence2.syndromicsurveillance.org/nssp_essence/usersguide/algorithms/TermBasedAlerts.jsp. You can pull word alerts from ESSENCE2 via the Alert List API. Term alert API URLs need to be defined in RStudio and are not populated in the ESSENCE2 user interface. This API requires the following parameters:
geo_system: region, facility, or regionFacilitystart_date: start date in ddMMMyyy formatend_date: end date in ddMMMyyy format
Optional query parameters include:
- filters: regions or facilities to limit to (defaults to all if not specified)
- show_stop_words: Boolean indicator of whether or not standard stop words should be removed (false by default)
- show_syndromic_words: Boolean indicator of whether or not to show syndromic terms used in syndrome definition queries (false by default)
- show_ignored_words: Boolean indicator of whether or not ignored words should be removed (false by default).
NSSP staff with admin access will be able to add and manage additional requested stop words in a list of “ignored” terms. This list will be modified over time to optimize the identification of informative terms and word pairs.
An example of an API URL only including required parameters is shown below. Fields in the corresponding alert table include term ID, date of the alert, data source ID and name, geography system, region (or “county”), anomalous term or word pair, detector algorithm, alert color ID (red = 3), frequency of occurrence, and expected values based on the 30-day baseline used. Note: Term-based alerts are only available for data from the last 7 days.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/alerts/wordAlerts?geo_system=region&start_date=7Sep2024&end_date=8Sep2024"
api_data <- get_api_data(url) %>%
pluck("termAlerts")ℹ Success: (200) OKThis second example demonstrates how users should form the URL with optional parameters; in this case example, the parameters filter to Broward County, FL, and include syndromic terms.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/alerts/wordAlerts?geo_system=region&start_date=27Dec2022&end_date=31Dec2022&filters=fl_broward&show_syndromic_words=true"
api_data <- get_api_data(url) %>%
pluck("termAlerts")ℹ Success: (200) OKTime Series Data Table with Stratified Alerts (Added August 2020)
Before this update, there was not an efficient way of pulling in daily stratified alerts. This update gave users the ability to pull historic alerts across stratifications in a long, tabular format. This functionality is only available from ESSENCE2: https://essence2.syndromicsurveillance.org/. For example, a user could choose multiple CCDD categories, Has been Emergency = “Yes”, CCDD Category for “As Percent Parameter”, and within the time series interface choose a geography level such as Hospital HHS Region for “Across Graphs Stratification” and CCDD Category for “Within Graph Stratification”. After selecting these configurations, you will need to select the “Update” button in the user interface to generate the stratified time series and API URL for the corresponding data table. Additionally, if “As Percent Parameter” is specified, the data pulled into RStudio will contain p-values and alert indicators specific to both counts and percentages. Indicators specific to counts are specified with “_dataCount” tags in the column names.
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/timeSeries?endDate=31Aug2024&ccddCategory=cdc%20pneumonia%20ccdd%20v1&ccddCategory=cdc%20coronavirus-dd%20v1&ccddCategory=cli%20cc%20with%20cli%20dd%20and%20coronavirus%20dd%20v2&percentParam=ccddCategory&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=31Dec2023&timeResolution=daily&hasBeenE=1&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TimeSeries&stratVal=ccddCategory&multiStratVal=geography&graphOnly=true&numSeries=3&graphOptions=multipleSmall&seriesPerYear=false&nonZeroComposite=false&removeZeroSeries=true&startMonth=January&stratVal=ccddCategory&multiStratVal=geography&graphOnly=true&numSeries=3&graphOptions=multipleSmall&seriesPerYear=false&startMonth=January&nonZeroComposite=false"
api_data <- get_api_data(url) %>%
pluck("timeSeriesData")ℹ Success: (200) OKNo encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.glimpse(api_data)Rows: 7,350
Columns: 21
$ date <chr> "2023-12-31", "2024-01-01", "2024-01-02", "…
$ count <dbl> 2.866, 2.725, 2.807, 2.643, 2.717, 2.566, 2…
$ expected <dbl> 2.436, 2.452, 2.462, 2.480, 2.485, 2.495, 2…
$ levels <dbl> 0.016, 0.034, 0.033, 0.084, 0.095, 0.186, 0…
$ colorID <int> 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ color <chr> "yellow", "yellow", "yellow", "blue", "blue…
$ altText <chr> "Data: Date: 31Dec23, Level: 0.016, Count: …
$ details <chr> "/nssp_essence/servlet/DataDetailsServlet?g…
$ graphType <chr> "percent", "percent", "percent", "percent",…
$ dataCount <dbl> 532, 542, 651, 599, 596, 535, 440, 345, 596…
$ expected_dataCount <dbl> 496.731, 501.852, 504.464, 508.036, 506.571…
$ levels_dataCount <dbl> 0.053, 0.100, 0.011, 0.010, 0.007, 0.064, 0…
$ colorID_dataCount <int> 1, 1, 2, 3, 3, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1…
$ color_dataCount <chr> "blue", "blue", "yellow", "red", "red", "bl…
$ allCount <dbl> 18564, 19891, 23188, 22662, 21936, 20846, 1…
$ lineLabel <chr> "CDC Pneumonia CCDD v1 - Region 1", "CDC Pn…
$ title <chr> "CDC Pneumonia CCDD v1 - Region 1", "CDC Pn…
$ ccddCategory_id <chr> "CDC Pneumonia CCDD v1", "CDC Pneumonia CCD…
$ ccddCategory_display <chr> "CDC Pneumonia CCDD v1", "CDC Pneumonia CCD…
$ hospitaldhhsregion_id <chr> "Region I", "Region I", "Region I", "Region…
$ hospitaldhhsregion_display <chr> "Region 1", "Region 1", "Region 1", "Region…names(api_data) [1] "date" "count"
[3] "expected" "levels"
[5] "colorID" "color"
[7] "altText" "details"
[9] "graphType" "dataCount"
[11] "expected_dataCount" "levels_dataCount"
[13] "colorID_dataCount" "color_dataCount"
[15] "allCount" "lineLabel"
[17] "title" "ccddCategory_id"
[19] "ccddCategory_display" "hospitaldhhsregion_id"
[21] "hospitaldhhsregion_display"A benefit of pulling the data and alerts in this manner is that you can create customized figures with ggplot or plotly that can be incorporated into static or interactive R Markdown reports:
hhs_region_data <- api_data %>%
select(
date,
hhs_region = hospitaldhhsregion_display,
ccdd_category = ccddCategory_display,
percent = count,
color
) %>%
mutate(
date = as.Date(date),
hhs_region = factor(hhs_region, levels = paste("Region", 1:10))
) %>%
filter(ccdd_category == "CLI CC with CLI DD and Coronavirus DD v2") %>%
arrange(date, hhs_region)
date_range <- paste(format(min(hhs_region_data$date), "%B %d, %Y"), "to", format(max(hhs_region_data$date), "%B %d, %Y"))
ggplot(hhs_region_data, aes(x = date, y = percent)) +
geom_line(linewidth = 0.7, color = "#046C9A") +
geom_point(data = subset(hhs_region_data, color == "red"), color = "red", size = 0.5) +
geom_point(data = subset(hhs_region_data, color == "yellow"), color = "yellow", size = 0.5) +
theme_bw() +
labs(
title = paste("CLI v2 by HHS Region:", date_range),
x = "Date",
y = "Percent of ED Visits"
) +
facet_wrap(facets = ~hhs_region, nrow = 2, scales = "free_x") +
scale_y_continuous(
limits = c(0, NA),
expand = expansion(c(0, 0.1))
) +
theme(
strip.background = element_blank(),
strip.text = element_text(face = "bold"),
panel.spacing = unit(1, "lines")
)
Capability of Stratifying by FIPS Code (Added July 2020)
ESSENCE2 includes facility county FIPS and patient county FIPS as available query fields (both are technically approximations since ESSENCE regions are populated by ZIP codes). Users may choose FIPS codes as a row (or column) field in the table builder. The following example assumes that a user has chosen the Facility Location (Full Details) data source, the CDC Coronavirus-DD v1, CDC Pneumonia CCDD v1, and Coronavirus-like illness (CLI) CC with CLI DD and Coronavirus DD v2 CCDD categories, “Yes” for “Has Been Emergency”, “CC and DD Category” for “As Percent Query”, and all counties within their state for Facility County FIPS Approximation. By selecting Date and Facility County FIPS Approximation for row fields and CC and DD Category for column fields, the API URL generated in the user interface will have the following structure:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/tableBuilder/csv?endDate=31Dec2022&facilityfips=...&percentParam=ccddCategory&datasource=va_hosp&startDate=03Oct2022&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TableBuilder&ccddCategory=cdc%20coronavirus-dd%20v1&ccddCategory=cdc%20pneumonia%20ccdd%20v1&ccddCategory=cli%20cc%20with%20cli%20dd%20and%20coronavirus%20dd%20v2&geographySystem=hospital&detector=nodetectordetector&timeResolution=daily&hasBeenE=1&rowFields=timeResolution&rowFields=facilityfips&columnField=ccddCategory"After the specifying endDate, all facility FIPS codes will be defined with the following syntax: “&facilityfips=fipscode1&facilityfips=fipscode2&…&facilityfips=fipscodeN&”. Note: Currently, users need to manually insert “&refValues=false” after specifying facilityfips as a row field in order to pull in the actual codes instead of the county names. Define this URL and API pull as follows:
url <- "https://essence2.syndromicsurveillance.org/nssp_essence/api/tableBuilder/csv?endDate=31Dec2022&facilityfips=...&percentParam=ccddCategory&datasource=va_hosp&startDate=03Oct2022&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TableBuilder&ccddCategory=cdc%20coronavirus-dd%20v1&ccddCategory=cdc%20pneumonia%20ccdd%20v1&ccddCategory=cli%20cc%20with%20cli%20dd%20and%20coronavirus%20dd%20v2&geographySystem=hospital&detector=nodetectordetector&timeResolution=daily&hasBeenE=1&rowFields=timeResolution&rowFields=facilityfips&refValues=false&columnField=ccddCategory"
api_data <- get_api_data(url, fromCSV = TRUE)Tips and Tricks
ESSENCE API URL Length
Occasionally, the API URL you created for the ESSENCE query will be very, VERY long. For example, you might have reason to explicitly include all facilities, all counties, or all ZIP codes for a site. In this situation, the character length of your URL might be too long to assign to the url <- object as shown in the preceding examples. When this occurs, you can split the URL into two or more strings when creating objects, and then pull them together later. In the code chunk shown below, we start with one long URL (use your imagination here) and break it into two pieces. Then, to create the object URL, we join them using the paste0() function. Unlike the paste() function, paste0() does not separate the combined character strings with a space. This final URL can be passed to ESSENCE along with your credentials to return your results. As a rule of thumb, do not exceed 20 lines of characters per URL.
url1 <- "https://essence.syndromicsurveillance.org/nssp_essence/api/very_very_very_long_URL_very_long_use_your_imagination_here...."
url2 <- "still_going_even_longer_here..........."
url3 <- paste0(url1, url2)
# Resulting URL
print(url)[1] "https://essence2.syndromicsurveillance.org/nssp_essence/api/timeSeries?endDate=31Aug2024&ccddCategory=cdc%20pneumonia%20ccdd%20v1&ccddCategory=cdc%20coronavirus-dd%20v1&ccddCategory=cli%20cc%20with%20cli%20dd%20and%20coronavirus%20dd%20v2&percentParam=ccddCategory&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=31Dec2023&timeResolution=daily&hasBeenE=1&medicalGroupingSystem=essencesyndromes&userId=2362&aqtTarget=TimeSeries&stratVal=ccddCategory&multiStratVal=geography&graphOnly=true&numSeries=3&graphOptions=multipleSmall&seriesPerYear=false&nonZeroComposite=false&removeZeroSeries=true&startMonth=January&stratVal=ccddCategory&multiStratVal=geography&graphOnly=true&numSeries=3&graphOptions=multipleSmall&seriesPerYear=false&startMonth=January&nonZeroComposite=false"Dynamically Set Start and End Dates
For reports run weekly or daily, you can automate the start and end dates rather than change the dates in the API URL manually before knitting. There are multiple ways to do this. You may split the URL into three pieces in a similar fashion to how the URL in the previous example was split. Or you may use str_extract() and str_replace() to substitute the appropriate dates. For example, if the report is based on the most recent 90 days, the start and end date can be auto-determined by using base R’s Sys.Date() and format() to ensure appropriate date formatting. format(Sys.Date(), "%d%b%Y") will give today’s date, 03Jan2023, while format(Sys.Date() - 89, "%d%b%Y") will give the start date of the recent 90 day period, 06Oct2022. You can insert the dates by splitting the URL can be split into three pieces and then pasting back together:
end_date <- format(Sys.Date() - 4, "%d%b%Y")
start_date <- format(Sys.Date()- 89 - 4, "%d%b%Y")
url <- paste0("https://essence.syndromicsurveillance.org/nssp_essence/api/timeSeries/graph?endDate=", end_date, "&medicalGrouping=injury&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=", start_date, "&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries&graphTitle=National%20-%20Injury%20Syndrome%20Daily%20Counts&xAxisLabel=Date&yAxisLabel=Count")
api_png <- get_api_graph(url)ℹ Success: (200) OKAdditionally, the URL start and end dates can remained fixed by setting up the code to simply extract and replace the old dates with new dates by using the change_dates() function from the Rnssp package. Please note that this is a new, preferred approach rather than manual coding via regular expressions…
url <- "https://essence.syndromicsurveillance.org/nssp_essence/api/timeSeries/graph?endDate=30Jun2021&medicalGrouping=injury&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=1Jun2021&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries&graphTitle=National%20-%20Injury%20Syndrome%20Daily%20Counts&xAxisLabel=Date&yAxisLabel=Count"
end_date_new <- Sys.Date() - 4
start_date_new <- Sys.Date() - 89 - 4
url_new <- change_dates(url, start_date = start_date_new, end_date = end_date_new)
url_new[1] "https://essence.syndromicsurveillance.org/nssp_essence/api/timeSeries/graph?endDate=29Nov2024&medicalGrouping=injury&percentParam=noPercent&geographySystem=hospitaldhhsregion&datasource=va_hospdreg&detector=probrepswitch&startDate=1Sep2024&timeResolution=daily&medicalGroupingSystem=essencesyndromes&userId=455&aqtTarget=TimeSeries&graphTitle=National%20-%20Injury%20Syndrome%20Daily%20Counts&xAxisLabel=Date&yAxisLabel=Count"Additional Resources
The preceding examples will help you pull data into your RStudio environment, but the next steps are really up to you. If you are unfamiliar with R and RStudio, here are some open-source resources to move your analysis forward.
- Rnssp Official Online Documentation
- Rnssp RMD Templates
- Rnssp Vignettes
- Rnssp Shiny Apps
- R for Data Science
- R for Data Science (2e)
- R Graphics Cookbook, 2nd Edition
- R Markdown: The Definitive Guide
- Text Mining with R
- Supervised Machine Learning for Text Analysis in R
- Modern R with Tidyverse
- R for Applied Epidemiology and Public Health
- Applied Epi
- A data.table and dplyr tour
- Time Series Analysis
- Comprehensive Guide to Using Quarto