Baseline assessment of The Workbench (TWB)

[摘要] The results of scientific data analyses are under increasing scrutiny, particularly when analysis results have an economic impact. As many analysis and processing tasks are repetitive in nature, there have been attempts to automate these tasks using the concept of a scientific workflow, where discrete data processing units are linked together in a combination of serial and parallel steps.A common means of implementing a scientific workflows uses command-line scripts, in a variety of languages, to process data: the outputs of one script feed the inputs of another. The scripting approach, however, is fragile: the inner workings of scripts are not always obvious or well-defined, scripts are often dependent on particular computer configurations, rarely is there governance surrounding script development, maintenance and deployment, and there is often no record of data provenance.The Environmental Information Systems (EIS) group, within CSIRO Land and Water, has developed a formal scientific workflow tool, "The WorkBench" (TWB), based on Microsoft's Trident workflow engine. Using Trident to manage workflow execution means that workflows can be re-run easily, and re-run by staff with less technical background; inputs to the processing steps ("activities")have data provenance recorded and activities and workflows can be transported between TWB installations. Trident also offers a platform to enable workflows to be run on High-Performance Computing IT infrastructure.TWB offers a suite of models and workflow processing steps aimed primarily at hydrologic data processing. To date, TWB has been used in the Water for a Health Country (WfHC) projects: Australian Water Resources Assessments, Catchment Yield Water Estimation Tool, Hydrological Geofabric, Risk Management Framework, and River Management for Multiple Outcomes.To ascertain where development efforts should be directed within the TWB project, a one-day workshop was held at CSIRO Land and Water (Black Mountain) on 26 July 2012. Participants included the TWB project staff, representatives from the WfHC project groups, and TWB users. The TWB team began the workshop with presentations on the current status of TWB: TWB's data handling capability, software development of TWB and governance of software development, documentation and training, version management and identification of potential TWB users and projects that could benefit from using TWB. The afternoon session sought to elicit more detail on each topic. The session and further discussions resulted in a set of recommendations for developing TWB to align more closely with the project groups' aims. The most important recommendations were as follows:•Develop TWB as a core platform for data processing, including improving ease-of-deployment and ease-of-use•Use continuous integration to deliver TWB software frequently and more responsively.•Govern, more formally, TWB's development, so as to make best use of development resources•Identify common TWB provenance use-cases and make provenance easier to access and understand•Add the ability for TWB to process data types common in the hydrology domain•Improve and strategically-target documentation and training to reduce TWB's steep learning curve •Improve TWB's ability to track and use different versions of activities•Build a community of practice around TWB to encourage potential TWB users to adopt the platform

[发布日期] 2012-08-15 [发布机构] CSIRO

[学科分类] 地球科学(综合)