If you missed the Workshop on Advanced Software Development Tools and Techniques held last July (I did---didn't even know it existed 'til December), there's a summary page with some interesting links. (See also this post about a stack analysis debugging tool developed at Livermore that collapses similar process states together to reduce the cognitive load on people programming massively parallel machines.)