2004 · 2005 · 2006 · 2007 · 2008 · 2009
2010 · 2011 · 2012 · 2013 · 2014 · 2015 · 2016 · 2017 · 2018 · 2019
2020 · 2021

Alpha Version of Software Tools in JavaScript

I have to fix two bugs in the examples, draw 91 diagrams, fill in an appendix on cognitive load, and then revise all 52,000 words, but the first draft is done. Feedback would be greatly appreciated: you can mail me or file issues in the book’s GitHub repository.

Title Pages Topic
Introduction 6
Systems Programming 20 Using callbacks to manipulate files and directories
Asynchronous Programming 22 Using promises to manage delayed computation
Unit Testing 12 Testing software piece by piece
File Backup 22 Archiving files with directory structure
Data Tables 16 Loading, saving, and manipulating tables efficiently
Pattern Matching 20 Using patterns to find things in data
Parsing Expressions 14 Turning text into code
Page Templates 18 Generating HTML pages from templates
Build Manager 22 Updating files that depend on other files
Layout Engine 24 Figuring out what goes where in a web page
File Interpolator 16 Managing source files that have been broken into pieces
Module Loader 16 Loading source files as modules
Style Checker 16 Checking that code conforms to style guidelines
Code Generator 14 Modifying code to track coverage and execution times
Documentation Generator 16 Generating documentation from comments embedded in code
Module Bundler 18 Turning many files into one
Package Manager 12 Getting and installing packages
Virtual Machine 16 Assembling and running low-level code
Debugger 18 Running programs under the control of a breakpointing debugger
Conclusion 2 Where we have been and where you could go next
Bibliography 2
License 4
Code of Conduct 2
Contributing 6
Cognition 2
Glossary 20
Links 6
Authors 6
Total 398

Pickled Carrots

By popular request, here is my spouse’s recipe for pickled carrots, which is derived from the one in Topp and Howard’s excellent The Complete Book of Small-Batch Preserving. Note that this is a canning recipe, not a cold pickle. Note also that the original recipe calls for only ¼ tsp of hot pepper flakes per jar, to which we say, “Bah.”

pickled carrots
Medieval ingredient metric
3 tbsp finely chopped fresh oregano or 1 tbsp (15 mL) dried 45 mL
1-3 tsp hot pepper flakes (per jar) 5-15 mL
2 small cloves garlic 2
1 lb peeled baby carrots 500 g
1 ½ cups white vinegar 375 mL
½ cup granulated sugar 125 mL
⅓ cup water 75 mL
1 tsp pickling salt 1 mL
  1. Remove the hot jars from the canner and add one garlic clove and your desired volume of chili flakes to each jar.

  2. Pack in the carrots (see picture) leaving 1cm (½ inch) of head space.

  3. Combine vinegar, sugar, water, and salt in a small saucepan and bring to a boil.

  4. Pour hot liquid over carrots (in jars) to within ½ inch of the top.

  5. Process for canning: 15 minutes for 500 mL jars.

A 2lb (approx. 1kg) bag of baby carrots will make 1750 mL of pickle. We usually use 250 mL wide-mouth jars—they make great gifts.

Steps in Order

Our book Research Software Engineering with Python has finally gone to the publisher. We will undoubtedly still have to make revisions, but the ideas and examples are complete. Our intended readers are captured in these three learner personas:

  • Amira Khan completed a master’s in library science five years ago and has since worked for a small aid organization. She did some statistics during her degree, and has learned some R and Python by doing data science courses online, but has no formal training in programming. Amira would like to tidy up the scripts, data sets, and reports she has created in order to share them with her colleagues. These lessons will show her how to do this.

  • Jun Hsu completed an Insight Data Science fellowship last year after doing a PhD in Geology and now works for a company that does forensic audits. He uses a variety of machine learning and visualization packages, and would now like to turn some of his own work into an open source project. This book will show him how such a project should be organized and how to encourage people to contribute to it.

  • Sami Virtanen became a competent programmer during a bachelor’s degree in applied math and was then hired by the university’s research computing center. The kinds of applications they are being asked to support have shifted from fluid dynamics to data analysis; this guide will teach them how to build and run data pipelines so that they can pass those skills on to their users.

We organized the book around a running example: the verification of Zipf’s Law using a set of classic English novels in an open, reproducible, and sustainable way. (People often conflate these three ideas, but they are distinct). To do that, we teach readers to do these things:

  1. Use the Unix shell to efficiently manage data and code.
  2. Organize small and medium-sized data science projects.
  3. Write Python programs that can be used on the command line.
  4. Use Git and GitHub to track and share work.
  5. Work productively in a small team where everyone is welcome.
  6. Use Make to automate complex workflows.
  7. Enable users to configure software without modifying it directly.
  8. Test software and know which parts have not yet been tested.
  9. Find, handle, and fix errors in code.
  10. Publish code and research in open and reproducible ways.
  11. Create Python packages that can be installed in standard ways.

The order is important because later skills depend on earlier ones, but also because we want people to be able to stop part way through and still have a workable research process. If you only get through the fourth chapter, for example, you’ll be able to back up your work, share it with others, and re-run analyses with a single command. Another chapter and you’ll be ready to collaborate in a team; one more after that, and you’ll be able to capture your workflows in re-runnable ways.

We don’t believe any one book can serve everyone’s needs, but we hope this one will help people who already know how to write a bit of code figure out what to learn next and what “done” looks like. The HTML version of the book will stay online for free and forever; we’ll advertise the printed and e-book versions as soon as they become available. If you find it useful, please let us know (and please also let us know about any errors or murky wording you stumble over).

This book has been over twenty years in the making; many thanks to my co-authors Damien Irving, Kate Hertweck, Luke Johnston, Joel Ostblom, and Charlotte Wickham for finally making it a reality. All royalties from the book will be donated to The Carpentries, an organization that teaches foundational coding and data science skills to researchers worldwide.

Concept Maps

In the second half of last year (and doesn’t it feel so good to call 2020 “last year”?) a group of us created and translated some concept maps for various topics in data science. What we have is online at https://github.com/rstudio/concept-maps/ under a Creative Commons license; we hope you find them useful.

Concent Map of Grouping and Summarizing

If you’d like to contribute, there are instructions on the site. And if you’d like to make contributing much easier, please take a look at this issue for http://diagrams.net: maintaining translations would be a lot easier if we could store alternate sets of text in drawings…