Pipeline visualisations

Description

A Pipeline object is automatically constructed as calls to ⁠make_*()⁠ are made. It stores the relationships between targets, dependencies, and sources.

Public fields

Methods

Public methods

• Pipeline$add_source_segment()

• Pipeline$add_recipe_segment()

• Pipeline$build()

• Pipeline$clean()

• Pipeline$touch()

• Pipeline$annotate()

• Pipeline$refresh()

• Pipeline$nomnoml()

• Pipeline$visnetwork()

• Pipeline$text_summary()

• Pipeline$print()

• Pipeline$save_visnetwork()

• Pipeline$save_nomnoml()

• Pipeline$save_text_summary()

• Pipeline$clone()

Method add_source_segment()

Add an edge to edges

Add any nodes in private$edges that are missing from private$nodes into private$nodes

Reconstruct Pipeline edges from Segment edges. Called primarily to update outofdateness

Add a pipeline segment corresponding to a make_with_source() call

Usage

Pipeline$add_source_segment(
  source,
  targets,
  dependencies,
  packages,
  envir,
  force
)

Arguments

Returns

The SegmentSource added to the Pipeline

Method add_recipe_segment()

Add a pipeline segment corresponding to a make_with_recipe() call

Usage

Pipeline$add_recipe_segment(
  recipe,
  targets,
  dependencies,
  packages,
  envir,
  force
)

Arguments

Returns

The SegmentRecipe added to the Pipeline

Method build()

Build all targets

Usage

Pipeline$build(quiet = getOption("makepipe.quiet"))

Arguments

Returns

self

Method clean()

Clean all targets

Usage

Pipeline$clean()

Returns

self

Method touch()

Touch all targets, updating file modification time to current time. This is useful when you know your targets are all up-to-date but makepipe doesn't (e.g. after a negligible change was made to your source code).

Usage

Pipeline$touch()

Returns

self

Method annotate()

Apply annotations to Pipeline

Usage

Pipeline$annotate(labels = NULL, notes = NULL)

Arguments

Method refresh()

Refresh Pipeline to check outofdateness

Usage

Pipeline$refresh()

Method nomnoml()

Display the pipeline with nomnoml

Usage

Pipeline$nomnoml(
  direction = c("down", "right"),
  arrow_size = 1,
  edge_style = c("hard", "rounded"),
  bend_size = 0.3,
  font = "Courier",
  font_size = 12,
  line_width = 3,
  padding = 16,
  spacing = 40,
  leading = 1.25,
  stroke = "#33322E",
  fill_arrows = FALSE,
  gutter = 5,
  edge_margin = 0
)

Arguments

Returns

self

Method visnetwork()

Display the pipeline with nomnoml

Usage

Pipeline$visnetwork(...)

Arguments

Returns

self

Method text_summary()

Display a text summary of the pipeline

Usage

Pipeline$text_summary()

Returns

self

Method print()

Display

Usage

Pipeline$print(...)

Arguments

Returns

self

Method save_visnetwork()

Save pipeline visNetwork

Usage

Pipeline$save_visnetwork(file, selfcontained = TRUE, background = "white", ...)

Arguments

Returns

self

Method save_nomnoml()

Save pipeline nomnoml

Usage

Pipeline$save_nomnoml(file, width = NULL, height = NULL, ...)

Arguments

Returns

self

Method save_text_summary()

Save a text summary of the pipeline

Usage

Pipeline$save_text_summary(file)

Arguments

Returns

self

Method clone()

The objects of this class are cloneable with this method.

Usage

Pipeline$clone(deep = FALSE)

Arguments

See Also

Other pipeline: pipeline-accessors, pipeline-vis

Segment

Description

A Segment object is automatically constructed and attached to the Pipeline when a call to ⁠make_*()⁠ is made. It stores the relationships between targets, dependencies, and sources.

Public fields

Active bindings

Methods

Public methods

• Segment$new()

• Segment$print()

• Segment$update_result()

• Segment$annotate()

• Segment$clone()

Method new()

Initialise a new Segment

Usage

Segment$new(
  id,
  targets,
  dependencies,
  packages,
  envir,
  force,
  executed,
  result,
  execution_time
)

Arguments

Method print()

Printing method

Usage

Segment$print()

Method update_result()

Update the Segment with new execution information

Usage

Segment$update_result(executed, execution_time, result)

Arguments

Method annotate()

Apply annotations to Segment

Usage

Segment$annotate(label = NULL, note = NULL)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

Segment$clone(deep = FALSE)

Arguments

See Also

Other segment: SegmentRecipe, SegmentSource

Segment

Description

A Segment object is automatically constructed and attached to the Pipeline when a call to ⁠make_*()⁠ is made. It stores the relationships between targets, dependencies, and sources.

Super class

makepipe::Segment -> SegmentRecipe

Public fields

Methods

Public methods

• SegmentRecipe$new()

• SegmentRecipe$update_result()

• SegmentRecipe$execute()

• SegmentRecipe$clone()

Method new()

Initialise a new Segment

Usage

SegmentRecipe$new(
  id,
  recipe,
  targets,
  dependencies,
  packages,
  envir,
  force,
  executed,
  result,
  execution_time
)

Arguments

Method update_result()

Update the Segment with new execution information

Usage

SegmentRecipe$update_result(executed, execution_time, result)

Arguments

Method execute()

Execute the Segment

Usage

SegmentRecipe$execute(envir = NULL, quiet = getOption("makepipe.quiet"), ...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

SegmentRecipe$clone(deep = FALSE)

Arguments

See Also

Other segment: SegmentSource, Segment

Segment

Description

A Segment object is automatically constructed and attached to the Pipeline when a call to ⁠make_*()⁠ is made. It stores the relationships between targets, dependencies, and sources.

Super class

makepipe::Segment -> SegmentSource

Public fields

Methods

Public methods

• SegmentSource$new()

• SegmentSource$update_result()

• SegmentSource$execute()

• SegmentSource$clone()

Method new()

Initialise a new Segment

Usage

SegmentSource$new(
  id,
  source,
  targets,
  dependencies,
  packages,
  envir,
  force,
  executed,
  result,
  execution_time
)

Arguments

Method update_result()

Update the Segment with new execution information

Usage

SegmentSource$update_result(executed, execution_time, result)

Arguments

Method execute()

Execute the Segment

Usage

SegmentSource$execute(envir = NULL, quiet = getOption("makepipe.quiet"), ...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

SegmentSource$clone(deep = FALSE)

Arguments

See Also

Other segment: SegmentRecipe, Segment

Parameters for make-like functions

Description

Parameters for make-like functions

Arguments

Register objects to be returned from make_with_source

Description

It is sometimes useful to have access to certain objects which are generated as side-products in a source script which yields as a main-product one or more targets. Typically these objects are used for checking that the targets were produced as expected.

Usage

make_register(value, name, quiet = FALSE)

Arguments

Value

value invisibly

Examples

## Not run: 
  # Imagine this is part of your source script:
  x <- readRDS("input.Rds")
  x <- do_stuff(x)
  chk <- do_check(x)
  make_register(chk, "x_check")
  saveRDS(x, "output.Rds")

  # You will have access to `chk` in your pipeline script:
  step_one <- make_with_source(
    "source.R",
    "output.Rds",
    "input.Rds",
  )
  step_one$result$chk

## End(Not run)

Create a pipeline using roxygen tags

Description

Instead of maintaining a separate pipeline script containing calls to make_with_source(), you can add roxygen-like headers to the .R files in your pipeline containing the ⁠@makepipe⁠ tag along with ⁠@targets⁠, ⁠@dependencies⁠, and so on. These tags will be parsed by make_with_dir() and used to construct a pipeline. You can call a specific part of the pipeline that has been documented in this way using make_with_roxy().

Usage

make_with_dir(
  dir = ".",
  recursive = FALSE,
  build = TRUE,
  envir = new.env(parent = parent.frame()),
  quiet = getOption("makepipe.quiet")
)

make_with_roxy(
  source,
  envir = new.env(parent = parent.frame()),
  quiet = getOption("makepipe.quiet"),
  build = TRUE
)

Arguments

Details

Other than ⁠@makepipe⁠, which is used to tell whether a given script should be included in the pipeline, the tags recognised mirror the arguments to make_with_source(). In particular,

• ⁠@targets⁠ and ⁠@dependencies⁠ are for declaring inputs and outputs, the expected format is a comma separated list of strings like ⁠@targets "out1.Rds", "out2.Rds"⁠ but R code like ⁠@targets file.path(DIR, "out.Rds")⁠ (evaluated in envir) works too

• ⁠@packages⁠ is for declaring the packages that the targets depend on, the expected format is ⁠@packages pkg1 pkg2 etc⁠

• ⁠@force⁠ is for declaring whether or not execution should be forced, the expected format is a logical like TRUE or FALSE

See the getting started vignette for more information.

Value

A Pipeline object

See Also

Other make: make_with_recipe(), make_with_source()

Examples

## Not run: 
# Create a pipeline from scripts in the working dir without executing it
p <- make_with_dir(build = FALSE)
p$build() # Then execute it yourself

## End(Not run)

Make targets out of dependencies using a recipe

Description

Make targets out of dependencies using a recipe

Usage

make_with_recipe(
  recipe,
  targets,
  dependencies = NULL,
  packages = NULL,
  envir = new.env(parent = parent.frame()),
  quiet = getOption("makepipe.quiet"),
  force = FALSE,
  label = NULL,
  note = NULL,
  build = TRUE,
  ...
)

Arguments

Value

A Segment object containing execution metadata.

See Also

Other make: make_with_dir(), make_with_source()

Examples

## Not run: 
# Merge files in fresh environment if raw data has been updated since last
# merged
make_with_recipe(
  recipe = {
    dat <- readRDS("data/raw_data.Rds")
    pop <- readRDS("data/pop_data.Rds")
    merged_dat <- merge(dat, pop, by = "id")
    saveRDS(merged_dat, "data/merged_data.Rds")
  },
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds")
)

# Merge files in current environment if raw data has been updated since last
# merged. (If recipe executed, all objects bound in source will be available
# in current env).
make_with_recipe(
  recipe = {
    dat <- readRDS("data/raw_data.Rds")
    pop <- readRDS("data/pop_data.Rds")
    merged_dat <- merge(dat, pop, by = "id")
    saveRDS(merged_dat, "data/merged_data.Rds")
  },
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds"),
  envir = environment()
)

# Merge files in global environment if raw data has been updated since last
# merged. (If source executed, all objects bound in source will be available
# in global env).
make_with_recipe(
  recipe = {
    dat <- readRDS("data/raw_data.Rds")
    pop <- readRDS("data/pop_data.Rds")
    merged_dat <- merge(dat, pop, by = "id")
    saveRDS(merged_dat, "data/merged_data.Rds")
  },
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds"),
  envir = globalenv()
)

## End(Not run)

Make targets out of dependencies using a source file

Description

Make targets out of dependencies using a source file

Usage

make_with_source(
  source,
  targets,
  dependencies = NULL,
  packages = NULL,
  envir = new.env(parent = parent.frame()),
  quiet = getOption("makepipe.quiet"),
  force = FALSE,
  label = NULL,
  note = NULL,
  build = TRUE,
  ...
)

Arguments

Value

A Segment object containing execution metadata.

See Also

Other make: make_with_dir(), make_with_recipe()

Examples

## Not run: 
# Merge files in fresh environment if raw data has been updated since last
# merged
make_with_source(
  source = "merge_data.R",
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds")
)


# Merge files in current environment if raw data has been updated since last
# merged. (If source executed, all objects bound in source will be available
# in current env).
make_with_source(
  source = "merge_data.R",
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds"),
  envir = environment()
)


# Merge files in global environment if raw data has been updated since last
# merged. (If source executed, all objects bound in source will be available
# in global env).
make_with_source(
  source = "merge_data.R",
  targets = "data/merged_data.Rds",
  dependencies = c("data/raw_data.Rds", "data/raw_pop.Rds"),
  envir = globalenv()
)

## End(Not run)

Check if targets are out-of-date vis-a-vis their dependencies

Description

Check if targets are out-of-date vis-a-vis their dependencies

Usage

out_of_date(targets, dependencies, packages = NULL)

Arguments

Value

TRUE if any of targets are older than any of dependencies or if any of targets do not exist; FALSE otherwise

Examples

## Not run: 
out_of_date("data/processed_data.Rds", "data/raw_data.Rds")

## End(Not run)

Access and interface with Pipeline.

Description

get_pipeline(), set_pipeline() and reset_pipeline() access and modify the current active pipeline, while all other helper functions do not affect the active pipeline

Usage

is_pipeline(pipeline)

set_pipeline(pipeline)

get_pipeline()

reset_pipeline()

Arguments

See Also

Other pipeline: Pipeline, pipeline-vis

Examples

## Not run: 
# Build up a pipeline from scratch and save it out
reset_pipeline()
# A series of `make_with_*()` blocks go here...
saveRDS(get_pipeline(), "data/my_pipeline.Rds")

# ... Later on we can read in and set the pipeline
p <- readRDS("data/my_pipeline.Rds")
set_pipeline(p)

## End(Not run)

Visualise the Pipeline.

Description

Produce a flowchart visualisation of the pipeline. Out-of-date targets will be coloured red, up-to-date targets will be coloured green, and everything else will be blue.

Usage

show_pipeline(
  pipeline = get_pipeline(),
  as = c("nomnoml", "visnetwork", "text"),
  labels = NULL,
  notes = NULL,
  ...
)

save_pipeline(
  file,
  pipeline = get_pipeline(),
  as = c("nomnoml", "visnetwork", "text"),
  labels = NULL,
  notes = NULL,
  ...
)

Arguments

Details

Labels and notes must be supplied as named character vector where the names correspond to the filepaths of nodes (i.e. targets, dependencies, or source scripts)

See Also

Other pipeline: Pipeline, pipeline-accessors

Examples

## Not run: 
# Run pipeline
make_with_source(
  "recode.R",
  "data/0 raw_data.R",
  "data/1 data.R"
)
make_with_source(
  "merge.R",
  c("data/1 data.R", "data/0 raw_pop.R"),
  "data/2 data.R"
)

# Visualise pipeline with custom notes
show_pipeline(notes = c(
  "data/0 raw_data.R" = "Raw survey data",
  "data/0 raw_pop.R" = "Raw population data",
  "data/1 data.R" = "Survey data with recodes applied",
  "data/2 data.R" = "Survey data with demographic variables merged in"
))

## End(Not run)