Import your data

Access the example datasets

An example dataset and data dictionary are included in every {soils} project. Access the .csv files in the data folder, or load the package and call the dataframes by name as shown below.

library(soils)
#> Found quarto! Replacing html engine...
head(washi_data, 5) |> knitr::kable()

head(data_dictionary, 5) |> knitr::kable()

Use washi_data and data_dictionary as templates when formatting your own data to use in {soils} functions and reports.

Data template

Your data must contain the below required columns and each soil measurement must be in its own column, as shown in washi_data.

dplyr::glimpse(washi_data)
#> Rows: 100
#> Columns: 42
#> $ year                   <int> 2023, 2022, 2022, 2022, 2023, 2022, 2023, 2022,…
#> $ sample_id              <chr> "23-WUY05-01", "22-RHM05-02", "22-ENR07-02", "2…
#> $ farm_name              <chr> "Farm 150", "Farm 085", "Farm 058", "Farm 061",…
#> $ producer_id            <chr> "WUY05", "RHM05", "ENR07", "ZTD04", "WUY05", "B…
#> $ field_name             <chr> "Field 01", "Field 02", "Field 02", "Field 03",…
#> $ field_id               <int> 1, 2, 2, 3, 3, 2, 1, 2, 1, 1, 1, 1, 2, 8, 2, 1,…
#> $ county                 <chr> "County 9", "County 18", "County 11", "County 1…
#> $ crop                   <chr> "Hay/Silage", "Green Manure", "Vegetable", "Her…
#> $ longitude              <int> -119, -123, -122, -120, -119, -117, -118, -117,…
#> $ latitude               <int> 49, 47, 47, 46, 49, 47, 49, 47, 48, 48, 46, 47,…
#> $ texture                <chr> "Clay Loam", "Sandy Loam", "Silt Loam", "Silt L…
#> $ bd_g_cm3               <dbl> 1.30, 0.88, 1.21, 1.37, 1.22, 1.14, 1.44, 1.24,…
#> $ pmn_lb_ac              <dbl> 67.13, 129.97, 122.17, 95.24, 111.35, 61.92, -7…
#> $ nh4_n_mg_kg            <dbl> 1.6, 21.6, 8.1, 13.8, 3.9, 12.4, 2.4, 12.4, 2.3…
#> $ no3_n_mg_kg            <dbl> 9.2, 6.1, 25.3, 16.9, 6.7, 4.3, 21.5, 7.4, 2.3,…
#> $ poxc_mg_kg             <int> 496, 571, 419, 424, 547, 235, 501, 480, 965, 10…
#> $ ph                     <dbl> 6.7, 5.9, 6.3, 6.8, 7.6, 5.5, 5.5, 5.9, 6.3, 6.…
#> $ ec_mmhos_cm            <dbl> 0.42, 0.05, 0.60, 2.18, 0.60, 0.81, 0.55, 0.34,…
#> $ k_mg_kg                <int> 498, 198, 294, 229, 273, 372, 289, 355, 253, 73…
#> $ ca_mg_kg               <int> 1380, 780, 1760, 3380, 2820, 1480, 1140, 2080, …
#> $ mg_mg_kg               <dbl> 145.2, 96.8, 266.2, 738.1, 193.6, 229.9, 133.1,…
#> $ na_mg_kg               <dbl> 16.1, 20.7, 20.7, 80.5, 13.8, 16.1, 23.0, 16.1,…
#> $ cec_meq_100g           <dbl> 7.8, 10.5, 13.0, 14.4, 10.1, 12.4, 12.9, 14.8, …
#> $ b_mg_kg                <dbl> 0.22, 0.09, 0.41, 0.72, 0.25, 0.18, 0.12, 0.21,…
#> $ cu_mg_kg               <dbl> 0.6, 0.4, 4.2, 1.1, 0.7, 1.0, 0.5, 1.4, 1.1, 0.…
#> $ fe_mg_kg               <int> 26, 28, 141, 37, 15, 64, 44, 85, 129, 31, 86, 3…
#> $ mn_mg_kg               <dbl> 1.5, 2.7, 4.1, 11.5, 1.7, 9.0, 4.4, 17.1, 9.9, …
#> $ s_mg_kg                <dbl> 4.29, 9.41, 26.73, 51.70, 3.29, 4.51, 9.13, 8.2…
#> $ zn_mg_kg               <dbl> 1.7, 0.8, 4.2, 2.4, 0.8, 0.5, 34.0, 0.9, 7.8, 0…
#> $ total_c_percent        <dbl> 1.85, 2.88, 1.68, 1.40, 1.65, 1.55, 2.25, 2.37,…
#> $ total_n_percent        <dbl> 0.16, 0.18, 0.14, 0.12, 0.16, 0.13, 0.15, 0.17,…
#> $ ace_g_protein_kg_soil  <dbl> 6.74, 21.50, 10.90, 5.53, 4.20, 10.30, 7.73, 6.…
#> $ sand_percent           <int> 44, 69, 11, 36, 64, 24, 80, 22, 62, 48, 80, 69,…
#> $ silt_percent           <int> 23, 21, 79, 51, 33, 62, 16, 57, 26, 45, 14, 27,…
#> $ clay_percent           <int> 33, 10, 10, 13, 3, 14, 4, 21, 12, 7, 6, 4, 10, …
#> $ min_c_96hr_mg_c_kg_day <dbl> 35.60, 30.00, 15.00, 67.50, 50.60, 25.50, 30.60…
#> $ p_olsen_mg_kg          <int> 15, 37, 73, 30, 8, 33, 27, 29, 40, 16, 19, 11, …
#> $ wsa_percent            <dbl> 88.5, 92.6, 91.3, 94.3, 84.6, 86.6, 86.9, 82.5,…
#> $ om_percent             <dbl> 4.5, 5.8, 2.4, 2.9, 6.7, 3.2, 27.0, 4.2, 7.9, 5…
#> $ toc_percent            <dbl> 1.85, 2.88, 1.68, 1.40, 1.53, 1.55, 2.25, 2.37,…
#> $ whc_in_ft              <dbl> 1.01, 1.08, 2.77, 1.93, 1.28, 2.25, 0.84, 2.14,…
#> $ inorganic_c_percent    <dbl> NA, NA, NA, NA, 0.12, NA, NA, NA, NA, NA, NA, N…

Your data must have the below required columns. However, only the columns in bold are required to have values. Put another way, your data must have these column names, even if every row is blank. Otherwise, the data-validation chunk in 01_producer-report.qmd or {soils} functions will error. For more details, see Data/dictionary mismatches.

• year <int> is used to select samples to be included in the report.

• sample_id <chr> must be unique throughout the dataset.

• farm_name <chr> is included at the top of the report. If blank, it is replaced with “Farm: producer_id”.

• producer_id <chr> is used to select samples to be included in the report.

• field_name <chr> must be unique across all fields for each producer and is displayed in tables and tool tips for maps. If blank, it is replaced with “Field field_id”.

• field_id <chr> must be unique across all fields for each producer.

• county <chr> is used to group and summarize samples from the same county as the producer. Can be blank.

• crop <chr> is used to group and summarize samples from the same crop as the producer. Can be blank.

• longitude <int> and latitude <int> are used to map each sample point using the {leaflet} package. Coordinates must be in decimal degrees using WGS 84 (aka EPSG:4326). If blank, the map will not be included.

• texture <chr> is used in the “physical” measurement group table. This column is not used in the texture triangle plot, which is instead created from percentage sand, silt, and clay measurement results.

• Each soil measurement must have its own column in the dataset and a corresponding row in the data dictionary, as shown in Dictionary template.

  • Measurement columns come after the required columns for easy selection in the tidy-long chunk.

Dictionary template

The data dictionary is used to group and order soil measurements, and to nicely format labels in tables and plots. The example data_dictionary contains UTF-8 encoded superscripts, subscripts, and special characters.

To properly encode your data dictionary as UTF-8, save it to the data folder as CSV UTF-8 (Comma delimited) (*.csv) in MS Excel.

dplyr::glimpse(data_dictionary)
#> Rows: 32
#> Columns: 4
#> $ measurement_group <chr> "Physical", "Physical", "Physical", "Physical", "Phy…
#> $ column_name       <chr> "texture", "sand_percent", "silt_percent", "clay_per…
#> $ abbr              <chr> "Texture", "Sand", "Silt", "Clay", "Bulk Density", "…
#> $ unit              <chr> "", "%", "%", "%", "g/cm³", "%", "in/ft", "%", "mg/k…

Your data dictionary must have the below required columns for every soil measurement included in your data.

• measurement_group <chr> determines how the soil measurements are grouped. The order in which these groups appear in the report is determined by the order they are listed within the data dictionary.

• column_name <chr> is used to join the dictionary with your data. Must exactly match the column names of the soil measurements in your dataset. The order in which these measurements appear within their measurement_group is determined by the order they are listed within the data dictionary.

• abbr <chr> and unit <chr> are how soil measurements are labeled in tables with the abbreviation as the column header and unit as a secondary spanning header.

Data/dictionary mismatches

Getting your data and dictionary in the proper format to render the report without errors will likely be the most difficult part of making {soils} work for your project.

The data-validation chunk and many {soils} functions include testthat::expect_contains() to fail early if required columns are missing from either the data or the dictionary. This early failure saves time by stopping as soon as the missing column or data/dictionary mismatch is identified. Additionally, these early failed tests provide the expected versus actual values, making it easier to correct the issue with minimal debugging.

Below are some common issues that arise and possible workarounds. These workarounds may make more sense after reviewing the Your data section that includes example errors and fixes.

Your data

Once your project data and dictionary files match the structure of the examples and are saved in the data folder, follow along with the changes in the code chunks in 01_producer-report.qmd. Code you will need to change are marked with the text “EDIT:”. Find all edit markers in the RStudio project with Ctrl + Shift + F to open the Find in Files wizard.

Below changes are for demonstration purposes only, the actual changes should be based on your data and dictionary.

# EDIT: Add your cleaned lab data to the data folder, using 'washi-data.csv' as
# a template.

# Load lab results
data <- read.csv(
  here::here("data/my-data.csv"),
  check.names = FALSE,
  encoding = "UTF-8",
  strip.white = TRUE
)

# EDIT: Add your data dictionary to the data folder, using 'data-dictionary.csv'
# as a template.

# Load data dictionary
dictionary <- read.csv(
  here::here("data/my-dictionary.csv"),
  check.names = FALSE,
  # Set encoding for using subscripts, superscripts, special characters
  encoding = "UTF-8",
  strip.white = TRUE
)

# EDIT: `washi_data` example has soil measurements in columns 12 - 42. Replace
# this column range with the column indices of the soil measurements in your
# dataset.

# Tidy data into long format and join with data dictionary
results_long <- data |>
  dplyr::mutate(
    dplyr::across(
      # EDIT: replace with the column range of your soil measurements
      12:26,
      as.numeric
    )
  ) |>
  tidyr::pivot_longer(
    # EDIT: replace with the column range of your soil measurements
    cols = 12:26,
    names_to = "measurement"
  )

Quitting from lines 94-115 [data-validation] (01_producer-report.qmd)
Error:
! names(data) (`actual`) isn't fully contained within c(required_cols, dictionary$column_name) (`expected`).
* Missing from `expected`: "tillage"
* Present in `expected`:   "year", "sample_id", "farm_name", "producer_id", "field_name", "field_id", "county", "crop", "longitude", ...

# OPTIONAL EDIT: If you have extra columns in `data`, add them to this vector.
required_cols <- c(
  "year",
  "sample_id",
  "farm_name",
  "producer_id",
  "field_name",
  "field_id",
  "county",
  "crop",
  "longitude",
  "latitude",
  "texture",
  "tillage"
)

# Check all column names in `data` are in the `required_cols` vector or
# `column_name` column of `dictionary`.
testthat::expect_in(names(data), c(required_cols, dictionary$column_name))

Quitting from lines 94-115 [data-validation] (01_producer-report.qmd)
Error:
! names(data) (`actual`) isn't fully contained within c(...) (`expected`).
* Missing from `expected`: "cec_meq_100g"
* Present in `expected`:   "year", "sample_id", "farm_name", "producer_id", "field_name", "field_id", "county", "crop", "texture", ...