Table of Contents · The Third Bit

2011-04-08

If all goes well, The Architecture of Open Source Applications will be available for purchase by the beginning of May. As a teaser, there’s a table of contents below the fold—we think it’s going to be a great book, and hope you’ll think so too.

Amy Brown & Greg Wilson: Introduction (15)
Russell Bryant: Asterisk (21)
- Critical Architectural Concepts (21)
  - Channels (21)
  - Channel Bridging (22)
  - Frames (23)
- Asterisk Component Abstractions (24)
  - Channel Drivers (24)
  - Dialplan Applications (26)
  - Dialplan Functions (27)
  - Codec Translators (27)
- Threads (28)
  - Network Monitor Threads (28)
  - Channel Threads (29)
- Call Scenarios (29)
  - Checking Voicemail (29)
  - Bridged Call (31)
- Final Comments (32)
James Crook: Audacity (35)
- Build Systems (36)
- wxWidgets Library-GUI (38)
- ShuttleGui Layer (39)
- The TrackPanel (40)
- PortAudio Library: Recording and Playback (41)
- BlockFiles (43)
- Scripting (45)
- Real-Time Effects (46)
- Summary (46)
Chet Ramey: The Bourne-Again Shell (49)
- Introduction (49)
  - Bash (49)
- Shell Syntactic Units and Primitives (51)
  - Shell Primitives (51)
  - Variables and Parameters (51)
  - The Shell Programming Language (52)
  - A Further Note (52)
- Input Processing (53)
  - Readline and Command Line Editing (53)
  - Non-interactive Input Processing (54)
  - Multibyte Characters (55)
- Parsing (55)
- Word Expansions (56)
  - Parameter and Variable Expansions (57)
  - And Many More (57)
  - Word Splitting (57)
  - Globbing (58)
  - Implementation (58)
- Command Execution (58)
  - Redirection (58)
  - Builtin Commands (59)
  - Simple Command Execution (60)
  - Job Control (60)
  - Compound Commands (61)
- Lessons Learned (61)
  - What I've Found is Important (61)
  - What I Would Have Done Differently (62)
- Conclusions (62)
Margo Seltzer & Keith Bostic: Berkeley DB (63)
- In the Beginning (63)
- Architectural Overview (65)
- The Access Methods: Btree, Hash, Recno, Queue (69)
- The Library Interface Layer (70)
- The Underlying Components (71)
- The Buffer Manager: Mpool (72)
  - The Mpool File Abstraction (72)
  - Write-ahead Logging (73)
- The Lock Manager:Lock (73)
  - Lock Objects (74)
  - The Conflict Matrix (74)
  - Supporting Hierarchical Locking (75)
- The Log Manager: Log (76)
  - Log Record Formatting (76)
  - Breaking the Abstraction (77)
- The Transaction Manager: Txn (78)
  - Checkpoint Processing (79)
  - Recovery (79)
- Wrapping Up (81)
Bill Hoffman & Ken Martin: CMake (83)
- CMake History and Requirements (83)
- How CMake is Implemented (85)
  - The CMake process (86)
  - CMake: The Code (87)
  - Graphical Interfaces (90)
  - Testing CMake (91)
- Lessons Learned (91)
  - Backwards Compatibility (91)
  - Language, language, language (92)
  - Plugins Did Not Work (92)
  - Reduce Exposed APIs (92)
Kim Moir: The Evolution of Eclipse Architecture (93)
- Early Eclipse (94)
  - Platform (94)
  - Java Development Tools (JDT) (102)
  - Plug-in Development Environment (PDE) (103)
- Eclipse 3.0: Runtime, RCP and Robots (103)
  - Runtime (103)
  - Rich Client Platform (RCP) (107)
- Eclipse 3.4: Brought to you by the letter p and the number 2 (108)
  - p2 Concepts (111)
- Eclipse 4.0 (112)
  - Modeled workbench (113)
  - Cascading Style Sheets (CSS) Styling (113)
  - Dependency Injection (115)
  - Application Services (118)
- Conclusion (118)
Chris Davis: Graphite (121)
- The Database Library---Storing Time-Series Data (121)
- The Back-end---A Simple Storage Service (122)
- The Front-end---Graphs On-Demand (123)
- Dashboards (124)
- An Obvious Bottleneck (124)
- Optimizing I/O (125)
- Keeping It Real-Time (126)
- Kernels, Caches, and Catastrophic Failures (126)
- Clustering (127)
  - A Brief Analysis of Clustering Efficiency (128)
  - Distributing Metrics in a Cluster (128)
- Design Reflections (129)
- Becoming Open Source (130)
Robert Chansler, Sanjay Radia, & Suresh Srinivas: The Hadoop Distributed File System (131)
- Introduction and Related Work (131)
- Architecture (132)
  - NameNode (132)
  - Image and Journal (132)
  - DataNodes (132)
  - HDFS Client (133)
  - CheckpointNode (134)
  - BackupNode (135)
  - Upgrades and File System Snapshots (135)
- File I/O Operations and Replica Management (136)
  - File Read and Write (136)
  - Block Placement (138)
  - Replication management (139)
  - Balancer (140)
  - Block Scanner (140)
  - Decommissioning (140)
  - Inter-Cluster Data Copy (141)
- Practice at Yahoo! (141)
  - Durability of Data (141)
  - Caring for the Commons (142)
  - Scaling and HDFS Federation (142)
- Lessons Learned (143)
- Acknowledgment (143)
C. Titus Brown & Rosangela Canino-Koening: Continuous Integration (145)
- The Landscape (145)
  - What Does Continuous Integration Software Do? (146)
  - External Interactions (148)
- Architectures (150)
  - Implementation Model: Buildbot (150)
  - Implementation Model: CDash (151)
  - Implementation Model: Jenkins (152)
  - Implementation Model: Pony-Build (153)
  - Build Recipes (155)
  - Trust (155)
  - Choosing a Model (155)
- The Future (156)
  - Concluding Thoughts (156)
  - Acknowledgements (157)
Emil Ivov: Jitsi (159)
- Designing Jitsi (159)
- Jitsi and the OSGi Framework (160)
- Building and Running a Bundle (162)
- Protocol Provider Service (164)
  - Operation Sets (165)
  - Accounts, Factories and Provider Instances (165)
- Media Service (167)
  - Capture, Streaming, and Playback (167)
  - Codecs (167)
  - Connecting with the Protocol Providers (168)
- UI Service (169)
- Lessons Learned (171)
  - Java Sound vs. PortAudio (171)
  - Video Capture and Rendering (172)
  - Video Encoding and Decoding (172)
  - Others (173)
- Acknowledgement (173)
Chris Lattner: LLVM (175)
- A Quick Introduction to Classical Compiler Design (176)
  - Implications of this Design (176)
- Existing Language Implementations (177)
- LLVM's Code Representation: LLVM IR (178)
  - Writing an LLVM IR Optimization (179)
- LLVM's Implementation of the Three-Phase Design (181)
  - LLVM IR is a Complete Code Representation (181)
  - LLVM is a Collection of Libraries (182)
- Design of the Retargetable LLVM Code Generator (184)
  - LLVM Target Description Files (184)
- Interesting Capabilities Provided by a Modular Design (186)
  - Choosing When and Where Each Phase Runs (186)
  - Unit Testing the Optimizer (187)
  - Automatic Testcase Reduction with BugPoint (188)
- Retrospective and Future Directions (188)
Dirkjan Ochtman: Mercurial (191)
- A Short History of Version Control (191)
  - Centralized Version Control (191)
  - Distributed Version Control (193)
- Data Structures (194)
  - Context (195)
  - A Solution: Revlogs (196)
  - The Three Revlogs (196)
  - The Working Directory (198)
- Versioning Mechanics (198)
  - Branches (198)
  - Tags (199)
- General Structure (199)
- Extensibility (200)
  - Writing Extensions (201)
  - Hooks (201)
- Lessons Learned (202)
Adam Marcus: The NoSQL Ecosystem (205)
- What's In a Name? (205)
  - SQL and the Relational Model (205)
  - NoSQL Inspirations (206)
  - NoSQL (207)
- NoSQL Data and Query Models (208)
  - Key-based NoSQL Data Models (208)
  - Graph Storage (210)
  - Complex Queries (210)
  - Transactions (210)
  - Schema-free Storage (211)
- Data Durability (211)
  - Single-server Durability (211)
  - Multi-server Durability (213)
- Scaling for Performance (213)
  - Do not Shard Until You Have To (214)
  - Sharding Through Coordinators (214)
  - Consistent Hash Rings (215)
  - Range Partitioning (217)
  - Which Partitioning Scheme to Use? (219)
- Consistency (219)
  - A little bit about CAP (220)
  - Strong Consistency (220)
  - Eventual Consistency (221)
- A Final Word (223)
- References (223)
Tarek Ziadé: Python Packaging (225)
- Introduction (225)
- The Burden of the Python Developer (226)
- The Current Architecture of Packaging (227)
  - Distutils Basics and Design Flaws (227)
  - Metadata and PyPI (229)
  - Architecture of PyPI (230)
  - Architecture of a Python Installation (234)
  - Setuptools, Pip and the Like (234)
  - What About Data Files ? (235)
- Improved Standards (236)
  - Metadata (236)
  - What's Installed? (238)
  - Architecture of Data Files (239)
  - PyPI Improvements (242)
- Implementation Details (244)
- Conclusion (246)
- References and Contributions (246)
Francesco Cesarini, Andy Gross, & Justin Sheehy: Riak and Erlang/OTP (247)
- An Abridged Introduction to Erlang (247)
- Process Skeletons (249)
- OTP Behaviors (250)
  - Introduction (250)
  - Generic Servers (251)
  - Starting Your Server (252)
  - Passing Messages (253)
  - Stopping the Server (255)
- Other Worker Behaviors (255)
  - Finite State Machines (255)
  - Event Handlers (256)
- Supervisors (257)
  - Supervisor Callback Functions (257)
  - Applications (260)
- Replication and Communication in Riak (260)
- Conclusions and Lessons Learnt (261)
  - Acknowledgments (261)
Simon Stewart: Selenium WebDriver (263)
- History (263)
- A Digression About Jargon (265)
- Architectural Themes (265)
  - Keep the Costs Down (266)
  - Emulate the User (266)
  - Prove the Drivers Work (266)
  - You Shouldn't Need to Understand How Everything Works (266)
  - Lower the Bus Factor (267)
  - Have Sympathy for a Javascript Implementation (267)
  - Every Call is an RPC Call (267)
  - Final Thought: This is Open Source (267)
- Coping with Complexity (268)
  - The WebDriver Design (269)
  - Dealing with the Combinatorial Explosion (271)
  - Flaws in the WebDriver Design (271)
- Layers and Javascript (272)
- The Remote Driver, and the Firefox Driver in Particular (275)
- The IE Driver (280)
- Selenium RC (284)
- Looking Back (286)
- Looking to the Future (287)
Eric Allman: Sendmail (289)
- A Fertile Field (289)
  - A Hodgepodge of Networks (289)
  - A Paucity of Technology (290)
  - A Plethora of Poorly Defined Standards (290)
  - A Pot of Glue (290)
- Design Principles (291)
  - Accept that One Programmer is Finite (291)
  - Don't Redesign User Agents (291)
  - Don't Redesign the Local Mail Store (291)
  - Make Sendmail Adapt to the World, Not the Other Way Around (292)
  - Change as Little as Possible (292)
  - Think About Reliability Early On (292)
- Development Phases (292)
  - Wave 1: delivermail (293)
  - Wave 2: sendmail 3, 4, and 5 (293)
  - Wave 3: the Chaos Years (294)
  - Wave 4: sendmail 8 (294)
  - Wave 5: the Commercial Years (295)
  - Whatever Happened to sendmail 6 and 7? (295)
- Design Decisions (296)
  - The Syntax of the Configuration File (296)
  - Rewriting Rules (296)
  - Using Rewriting to do Parsing (297)
  - Embedding SMTP and Queueing in sendmail (297)
  - The Implementation of the Queue (298)
  - Accepting And Fixing Bogus Input (299)
  - Configuration and the Use of M4 (299)
- Other Considerations (300)
  - A Word About Optimizing Internet Scale Systems (300)
  - Milter (300)
  - Release Schedules (301)
- Security (301)
- Evolution of Design (302)
  - Configuration Became More Verbose (302)
  - More Connections with Other Subsystems: Greater Integration (303)
  - Adaptation to a Hostile World (303)
  - Incorporation of New Technologies (304)
- What If I Did It Today? (304)
  - Things I Would Do Differently (304)
  - Things I Would Do The Same (305)
- Conclusions (306)
Roy Bryant & Andrés Lagar Cavilla: Snowflock (307)
- Introducing Snowflock (307)
- VM Cloning (308)
- SnowFlock's Approach (309)
- Architectural VM Descriptor (311)
- Parent-Side Components (311)
  - Memserve Process (311)
  - Multicasting with Mcdist (312)
  - Virtual Disk (313)
- Clone-Side Components (313)
  - Memtap Process (313)
  - Clever Clones Avoid Unnecessary Fetches (314)
- VM Cloning Man Page (314)
  - API Implementation (314)
  - Necessary Mutations> (315)
- Conclusion (316)
Audrey Tang: Social Calc (317)
- Introduction (317)
- WikiCalc (318)
- SocialCalc (321)
- Command Run-loop (323)
- Table Editor (325)
- Save Format (327)
- Rich-text Editing (328)
  - Types and Formats (329)
  - Rendering Wikitext (331)
- Real-time Collaboration (332)
  - Cross-browser Transport (332)
  - Conflict Resolution (335)
  - Remote Cursors (336)
- Lessons Learned (337)
  - A Chief Designer with a Clear Vision Worked Really Well (337)
  - Wikis for Project Continuity (337)
  - Embrace Time Zone Differences (338)
  - Optimize for Fun (338)
  - Drive development with Story Tests (338)
  - Open Source With CPAL (339)
Danielle Madeley: Telepathy (341)
- Components of the Telepathy framework (342)
- How Telepathy uses D-Bus (344)
  - Handles (346)
  - Discovering Telepathy Services (347)
  - Reducing D-Bus Traffic (347)
- Connections, Channels and Clients (349)
  - Connections (349)
  - Channels (349)
  - Requesting Channels, Channel Properties and Dispatching (350)
  - Clients (352)
- The Role of Language Bindings (353)
  - Asynchronous Programming (353)
  - Object Readiness (354)
- Robustness (355)
- Extending Telepathy: Sidecars (356)
- A Brief Look Inside a Connection Manager (356)
- Lessons Learned (357)
Alan Laudicina & Aaron Mavrinac: Thousand Parsec (359)
- Anatomy of a Star Empire (360)
  - Objects (361)
  - Orders (361)
  - Resources (362)
  - Designs (363)
- The Thousand Parsec Protocol (363)
  - Basics (364)
  - Players and Games (364)
  - Objects, Orders, and Resources (366)
  - Design Manipulation (366)
  - Server Administration (368)
- Supporting Functionality (369)
  - Server Persistence (369)
  - Thousand Parsec Component Language (369)
  - BattleXML (370)
  - Metaserver (370)
  - Single-Player Mode (370)
- Lessons Learned (372)
  - What Worked (373)
  - What Didn't Work (373)
  - Conclusion (373)
Cay Horstmann: The Violet UML Editor (375)
- Introducing Violet (375)
- The Graph Framework (376)
- Use of JavaBeans Properties (380)
- Long-Term Persistence (382)
- Java WebStart (384)
- Java 2D (384)
- No Swing Application Framework (386)
- Undo/Redo (387)
- Plugin Architecture (388)
- Conclusion (389)
Juliana Freire, David Koop, Emanuele Santos, Carlos Scheidegger, Claudio Silva, & Huy Vo: VisTrails (391)
- System Overview (392)
  - Workflows and Workflow-Based Systems (392)
  - Data and Workflow Provenance (393)
  - User Interface and Basic Functionality (394)
  - Project History (394)
- Inside VisTrails (395)
  - The Version Tree: Change-Based Provenance (395)
  - Workflow Execution and Caching (397)
  - Data Serialization and Storage (397)
  - Extensibility Through Packages (398)
  - VisTrails Packages and Bundles (399)
  - Passing Data as Modules (400)
- Components and Features (401)
  - Visual Spreadsheet (401)
  - Visual Differences and Analogies (403)
  - Querying Provenance (403)
  - Persistent Data (404)
  - Upgrades (405)
  - Sharing and Publishing Provenance-Rich Results (405)
- Lessons Learned (407)
  - Acknowledgments (407)
Berk Geveci & Will Schroeder: VTK (409)
- What Is VTK? (410)
- Architectural Features (410)
  - Core Features (410)
  - Representing Data (412)
  - Pipeline Architecture (413)
  - Rendering Subsystem (417)
  - Events and Interaction (419)
  - Summary of Libraries (420)
- Looking Back/Looking Forward (420)
  - Managing Growth (421)
  - Technology Additions (421)
  - Open Science (422)
  - Lessons Learned (422)
Richard Shimooka & David White: Battle For Wesnoth (425)
- Project Overview (426)
- Wesnoth Markup Language (427)
- Units in Wesnoth (430)
- Wesnoth's Multiplayer Implementation (431)
- Conclusion (432)