Task Files

Published:

Last Updated:

In a recent chat with Andy Chu, creator of Oils, he mentioned his use of “task files” for most shell usage. I hadn’t encountered that term before. The concept is simple—you type a set of shell functions in a single file and make them invocable interactively. Here’s a simple example:

#!/bin/sh
# tasks.sh

run_tests() {
  # ...
}

build() {
  # ...
}

deploy() {
  # ...
}

"$@"

To invoke the build command, for example, run ./tasks.sh build.1

Task files are a useful pattern which has been broadly documented (a web search for task files will surface numerous references going back many years), but the technique is new to me, and that means it will be new to others as well. So I’d like to record my initial perspectives and lessons learned—which might be better conceived as a set of aspirations. I’ll also share some work-in-progress tooling to facilitate documentation and discoverability of tasks.

First Thoughts and Lessons Learned

Thinking in Tools

At first, task files seemed like a different flavor of a pattern I’ve applied for years. Each of my projects have a scripts directory with the usual suspects:

scripts/
  build.sh
  deploy.sh
  format.sh
  lint.sh
  test.sh

On the surface, a task file simply unifies these scripts into a... file of tasks, but there’s more going on. Over time, each dedicated script accretes features. In mature projects, scripts often contain latent tools. By allowing external invocation of functions, the task file invites us to discover these tools and liberate them. While converting my scripts, I continue to find hidden tools and to wonder how I might create more. My perception is shifting away from coarse tasks to a more fine-grained tool approach. In this sense, I’ve begun to think tool files might be a better name than task files, but I won’t fight the established terminology.

Sharing Tools

When working on a project, I have a bad habit of developing a corpus of ad hoc tools for my own work that don’t make it into source control. It turns out colleagues do the same thing! Task files are an invitation to share these tools in one place. For that to happen well, tools need to be documented and discoverable (more on that later).

Beyond providing other contributors with useful tools, this is valuable knowledge sharing—I might not have known that a thing was possible without seeing someone else do it. I think this is akin to pair programming with someone who is highly skilled with their development tools—you can learn a lot just by observing. How much better would it be if you can benefit directly through access to a tool and its implementation for study? Of course, this is also possible by adding discrete scripts to a project, but task files arguably have a lower barrier to entry simply because there’s a single file that already exists—there’s one way to do it.

Anecdotally, a coworker was recently tasked with combing through a large dataset to find all unique instances of a value. They were going to search through each file by hand! They didn’t know that find . -type f -name '*.json' -print0 | xargs -0 -- jq .someValue | sort | uniq was an option. In the few seconds that it took to type and copy/paste the command, my coworker was spared hours(?) of tedious, error-prone, and unnecessary drudgery. It’s okay to not know something—it’s not okay to know and not share it. This is not to suggest that the time-saving tool would have existed prior to the need for it but rather that task files can be mechanism for building a culture of knowledge sharing (e.g., a known, living collection of data analysis tools).

Tool Incubator

It seems probable that a tool that starts its life in a task file might prove generally useful and be promoted to a system-wide tool (this has already happened once for me with the style tool from the task script I use to develop this site). Before that happens, the task file acts as an incubator—you can figure out what does and doesn’t work in the context of a real project with concrete needs.

Development Workflow

When each task is a dedicated script, there was likely a moment in time that someone decided to sit down and write that script. It may have evolved over time, but there was some conscious effort to create it for a (usually coarse-grain) purpose. I’m finding that task files can enable a more experimental, organic workflow where micro-tasks grow into tools over time. When I need to achieve some small project-related task, I don’t immediately reach for the interactive shell—opting to type the command into the task file instead. This has a few advantages:

Beyond the development workflow, there are similar time-of-use advantages over shell history:

A Work-In-Progress Task File Library

While experimenting with task files, an early insight was that they would be more useful for me and my team if the tasks were easier to discover and document. To that end, I’m developing a small tool to inject task files with an automated task discovery mechanism and help text.

All task files get an implicit list-tasks tool for free. From the task file for this site:

$ ./run.rc list-tasks
build - Build the site
compile-page - Compile a single page
deploy - Build and deploy the site
log - Log to standard error
serve - Serve the site at localhost:8000
style - Style text with the given options
timestamp - Output an HTML fragment with the current date-time
trim-prefix - Trim a specified prefix from the given string

Each task also gets -h, --help flags for free with a minimal default usage text or better text generated from a loosely structured help comment (also used to supply the “summary” for each task in the example above). For example, the following comment:

## usage: style [OPTIONS ...] TEXT ...
##
## summary: Style text with the given options
##
## options:
## -f, --fg COLOR
##     set the foreground color
##
## -b, --bg COLOR
##     set the background color
##
## -B, --bold
##     make the text bold
##
## -i, --italic
##     italicize the text
##
## -u, --underline
##     underline the text
##
## -r, --reverse
##     swap the foreground and background colors
##
## -s, --strike
##     strikethrough the text
fn style {
    # implementation ...
}

is used to provide:

$ ./run.rc style --help
USAGE: style [OPTIONS ...] TEXT ...

Style text with the given options

OPTIONS:
-h, --help
    display this help text

-f, --fg COLOR
    set the foreground color

-b, --bg COLOR
    set the background color

-B, --bold
    make the text bold

-i, --italic
    italicize the text

-u, --underline
    underline the text

-r, --reverse
    swap the foreground and background colors

-s, --strike
    strikethrough the text

There are four directives (all optional):

  1. usage provides a one-line synopsis of the tool’s interface
  2. summary distills the functionality of the tool in one line
  3. description is a free-form, multi-line field for extended description of the tool
  4. options is a free-form, multi-line listing of the flags supported by the tool (an entry for the -h, --help flag is inserted automatically as the first entry—in the style shown above)

The directives have to appear in the specified order with the line restrictions mentioned. Otherwise, the format is up to the author and will be printed verbatim. Any directive may be omitted.

Maybe someone will find this useful!

In Summary

Task files have already proven beneficial for me—unlocking a different (if only in degree) way of thinking about tools. If you’re a seasoned user of task files, I’d love to hear your experience. If this is all new to you, welcome to the club—I hope you’ll give them a try!

This is the first article on this site. I don’t anticipate adding a comment section, but feel free to send comments to will at this domain. I’ll be happy to share useful feedback and post attribution for corrections, and I’m always eager just to chat.

1If you’re new to shell scripting, the last line of the script is the key. The special variable $@ expands to the positional parameters provided by the user when the script is executed. The surrounding double quotes are critical to prevent field splitting (also known as word splitting). By passing a function name as the first positional parameter, we can effectively invoke the function from the outside.