Skip to content
/ bom Public

Tools to Identify and Work with Byte Order Marks in R

Notifications You must be signed in to change notification settings

hrbrmstr/bom

Repository files navigation

bom : Tools to Identify and Work with Byte Order Marks

Byte order marks (BOM) appear at the beginning of a file or buffer and provide information about the encoding of the contents. R provides facilities to work with files and connections with BOMs but there are situatons where these facilities are not sufficient. Tools are provided to identify the presence and type of byte order marks in files and R objects as well as remove them.

The following functions are implemented:

  • file_bom_type: Get BOM type (file)
  • file_has_bom: Test if a file has a BOM
  • raw_bom_type: Get BOM type (raw vector)
  • raw_has_bom: Test if a raw vector has a BOM

TODO

  • Convert to S3 methods
  • BOM removal functions
  • Mechanism to return a connection sans BOM or identify there is a BOM

Installation

devtools::install_git("https://gitlab.com/hrbrmstr/bom.git")
options(width=120)

There are some basic examples in the Usage section, but this may be a better illustration. Say you have a CSV file:

fil <- system.file("examples", "stop_times.txt", package="bom")

And, say you want to read it in with a more modern CSV reader:

library(readr)

df <- read_csv(fil)
## Parsed with column specification:
## cols(
##   `trip_id` = col_integer(),
##   arrival_time = col_time(format = ""),
##   departure_time = col_time(format = ""),
##   stop_id = col_integer(),
##   stop_sequence = col_integer(),
##   pickup_type = col_integer(),
##   drop_off_type = col_integer()
## )

Let's look at that file:

print(df, n=1)
## # A tibble: 64,827 × 7
##   `trip_id` arrival_time departure_time stop_id stop_sequence pickup_type drop_off_type
##       <int>       <time>         <time>   <int>         <int>       <int>         <int>
## 1     50000   29700 secs     29700 secs  120001            41           1             0
## # ... with 6.483e+04 more rows

Hrm…why are those backticks around trip_id? Isn't it just a regular string?

print(colnames(df)[1])
## [1] "trip_id"

It sure looks that way, but looks can be deceiving:

print(charToRaw(colnames(df)[1]))
##  [1] ef bb bf 74 72 69 70 5f 69 64

Those strange characters at the beginning are a byte order mark (BOM). We can test for it being there and work around it:

library(bom)

if (file_has_bom(fil)) {
  n <- switch(file_bom_type(fil), `UTF-8`=3, 2)
  df <- read_csv(readBin(fil, "raw", file.size(fil))[-(1:n)])
}
## Parsed with column specification:
## cols(
##   trip_id = col_integer(),
##   arrival_time = col_time(format = ""),
##   departure_time = col_time(format = ""),
##   stop_id = col_integer(),
##   stop_sequence = col_integer(),
##   pickup_type = col_integer(),
##   drop_off_type = col_integer()
## )
print(df, n=1)
## # A tibble: 64,827 × 7
##   trip_id arrival_time departure_time stop_id stop_sequence pickup_type drop_off_type
##     <int>       <time>         <time>   <int>         <int>       <int>         <int>
## 1   50000   29700 secs     29700 secs  120001            41           1             0
## # ... with 6.483e+04 more rows
charToRaw(colnames(df)[1])
## [1] 74 72 69 70 5f 69 64

Note that the built-in read.csv() can be used with encoding="UTF-8-BOM" and you can even use that encoding on non-binary connections, but you end up having to type convert and tibble convert that object so you're basically rewriting (badly) readr::read_csv().

Usage

library(bom)

# current verison
packageVersion("bom")
## [1] '0.1.0'
file_has_bom(system.file("examples", "stops.txt", package="bom"))
## [1] FALSE
file_has_bom(system.file("examples", "stop_times.txt", package="bom"))
## [1] TRUE
raw_has_bom(readBin(system.file("examples", "stop_times.txt", package="bom"), "raw", 4))
## [1] TRUE
file_bom_type(system.file("examples", "stops.txt", package="bom"))
## [1] NA
file_bom_type(system.file("examples", "stop_times.txt", package="bom"))
## [1] "UTF-8"
raw_bom_type(readBin(system.file("examples", "stop_times.txt", package="bom"), "raw", 4))
## [1] "UTF-8"

Test Results

library(bom)
library(testthat)

date()
## [1] "Sat Oct  1 09:26:55 2016"
test_dir("tests/")
## testthat results ========================================================================================================
## OK: 6 SKIPPED: 0 FAILED: 0
## 
## DONE ===================================================================================================================

About

Tools to Identify and Work with Byte Order Marks in R

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published