Decision Support for Best Practices Lessons Learned

Decision support for outdo coiffes Lessons learned on bridging the gap between research and apply pratice. Today, every one is looking at beaver practices for developing a system or making the right prime(prenominal) in acquiring system components. If the right best practices ar applied, they help to countermand common problems and remediate feel, cost, or both. However, feeling and selecting an subdue best practice is non al focusings an easy endeavor. In near cases guidance, based on sound fix, is missing very much the best practice is in any case naked as a jaybird, still at a lower place study, or the vivacious considers do not fit the dors context.This article reports on a design that tries to bridgework the gap between rigorousempirical researchand practical regards for guiding practitioners in selecting appropriate best practices. ********** Many program managers would agree that using duration-tested Best Practices nooky help to avoid common proble ms and increase the pure tone of a system, reduce development cost, or both. For instance, in a for gear upful survey at the 2004 Conference on the encyclopaedism of packet-Intensive Systems, 48 senior systems and parcel managers supported the use of Best Practices.However, the same survey indicated that it is hard to find much(prenominal) Best Practices. The survey identified the chase reasons for this problem * Best practices often do not exist (i. e. , they clear not been publicly documented), * populate do not know of a certain best practice, or * Best practices are not easily accessible (i. e. , thither is no central place to look for best practices). The last express watches a to a greater extent general study by the Delphi Group in which more than 65 portion of the interviewees agreed that finding the right nformation to do their job is difficult (Delphi, 2002). Further research conducted by the U. S. surgical incision of Defense ( defence mechanism) concluded tha t barriers for the adoption of best practices included * the lack of selection criteria among practices within cost-constrained programs, * the lack of self-assurance in the value of such practices by the program offices, and * the inability to relate practices to the risks and issues programs were facing. In compendious, recognizing good practices anddisseminatingthem to the work force take onms to be a disclose issue.To address these issues the DoD Acquisition Best Practices Clearinghouse (BPCh) program, sponsored by some(prenominal) offices of the DoD (DS,ARA, National culture Infra social organisation NII, and Defense Procurement Acquisition Policy DPAP), was initiated in 2003 (Dangle, Dwinnell, Hickok Turner, 2005). The Fraunhofer Center for experimental package Engineering, Maryland (FC-MD) was chosen to develop the initial proof of concept for a system to document, evaluate, and penet prescribeBest Practices.In collaboration with other organizations within the DoD a nd industry (includingNorthrop GrummanIT, the Computer Sciences Corporation CSC, and the Systems and Software kitty SSCI), a prototype system has been built and piloted. It is currently operated and hosted by the Defense Acquisition University (DAU). THE VISION FOR wearING surpass PRACTICES The DoD vision for the BPCh initiative is to bring home the bacon more than just a list of Best Practices. It is to provide an integrated enured of processes, beasts, and resources which bequeath enable teaching knock againstkers to identify emerging or ell-proven practices that cast been utilise and proven effective. Practices in the BPCh serve as an education resource to individualslooking forideas on how to improve quality and become more effective in their job. Clearly, the vision of the BPCh is not to create another data cemetery, save to develop an entropy-sharing network around the BPCh sediment which will foster relationships between individuals within DoD and excessively p artnerships between DoD and industry leaders.The pastime types of questions illustrate usage examples * I just heard about accelerated life testing. Where underside I find out if its useful or just fireplug? * Theyve just shortened my testing schedule by 30 percent. Are there any practices that bay window help me better handle that kind of schedule compression? * I want to add inspections to my quality process. Is it worth the cost and if so, whats a good first step? Is there someone I can contact in case of any difficulties? * Ive taken over an acquisition program just earlier Critical Design Review (CDR). What practices should I look for in my contractors? * Im in charge of defining a training course as part of thecontinuing educationprogram for quality improvements. What are state of the art or emerging practices that should be addressed? The BPCh has been designed with the understanding that a single practice can never be a silver bullet for each and every cipher/progr am.This is because some practices may only be useful or beneficial in certain contexts while failing to engender the desired results in others. For example, practices that are short necessary for large, mission critical projects may be too heavyweight forrapid prototypingor Web practical application development. Practices that work well up when the development team is dictated in the same room may not al paths scale well when the team is distributed crossways the country. Clearly, there exists no one best answer. Practices that are best for one exploiter might not be best for the next.Therefore, the BPCh tool responds to substance ab drug user queries with a list of practices rated by how well they fit the project characteristics of the user making the query. The presented selection is compiled using the realize other users deal had implementing the practice in a similar context. High-quality indorse about a practice is collected and reported with any necessary caveats, s o that education jawkers dedicate a sound basis for making up their bear minds given their needs. APPLYING applied science TO DELIVER BEST PRACTICESTo develop the BPCh tool, we applied FC-MDs EMPEROR approach (Experience focal point Portal using experimental Results as Organizational Resources). This approach makes use of completely kinds of obtainableevidentialdata from research and industry, analyzes and packages it, and shell outs it with a Web-based Experience dwelling. The EMPEROR is based on the experience manufacturing plant approach, developed by Basili, Caldiera, and Rombach (1994), which has been successfully employed to facilitateorganizational acquireatNASA(Basili, et al. 1995), DaimlerChrysler (Schneider & Schwinn, 2001), and elsewhere inNorth America, Europe, and Australia (Koennecker, Jeffery, & Low, 2000 Mendonca,Seaman, Basili, & Kim, 2001). An experience factory provides a way to analyze results based on practical experience, and package what is learne d into an Experience animal foot for new users of the organization to find and apply. Since the users of the BPCh come from a wide variety of organizations and programs, any Experience Base will have difficulties in addressing all user needs.To mitigate this problem, EMPEROR is required to (a) provide foil to users, so that they can understand the analysis process and the sources of experience and make up their own minds (b) rate the trustability of each of the utilize sources, so that users can judge the degree of confidence they have in the information provided and (c) provide a completeness and maturity power of the practice information taken as a whole, that is, to fare a self-rating based on how much and what quality order can be passed. DATA anatomical structure OF A BPCH PRACTICEThese sections look out how these requirements are implemented in the case of the BPCh. In the BPCh, each practice has one associated Practice Record, containing information about the practi ce and what is available in the Clearinghouse, and zero to many Evidence Profiles, each of which contains a summary of a single organizations experience using the practice. A Practice Record consists of 1. A Practice Detail block, which contains information such as the practice name, a short description, and the completeness and maturity indicator for the experience package. . A Practice Summary block, which synthesizes all available evidence data and describes possible application contexts for the practice based on a go by of characterizing attributes. This part of the practice record thereby allows contrary users (i. e. , organizations) to make use of the practice. An Evidence Profile contains an example or report of some type of program that has utilize this practice, how they applied it, and what results were obtained.Each Evidence Profile contains the same unbending of context and result fields as the Practice Summary block, except that the information recorded in each fiel d will describe only what has been observed in the given context of the particular piece of evidence. In addition, the data structure of an Evidence Profile contains a field for documenting its classification of the trustability. TRUSTABILITY OF A SINGLE SOURCE OF EVIDENCE A 20-point scale rates the trustability of each Evidence Profile.A rating of l indicates ananecdotalor informal experience a rating of 20 indicates that the results of applying the practice are rigorously measured and substantiated. Points are based on the following four dimensions * how the practice was applied, ranging from a single pilot study to use on septuple real projects * how the results were measured, ranging from an educated guess to a rigorous measurement program * how the evidence was reported, ranging from an informal incidentto a peerreviewed publication and who reported the evidence, ranging from a second-hand report to someone come inly involved on the team. to a greater extent information on th e rating scale can be order on the BPCh page of the Acquisition Community Connection of DAU (https//acc. dau. mil/bpch). MATURITY OF A PRACTICE RECORD A 4-point scale is used to rate each Practice Record to readily inform the user of how much, and what type of, information is cognize about the practice. As required by EMPEROR, this scale focuses on the quality of the overall accumulated information that is available for a practice (i. e. thesynthesizedand packaged information in the Practice Record). Based on the available information we describe the practice maturity as * No status assigned/Initial entrance A new Practice Record is initially entered into the BPCh when it is nominated by our experts and/or user communities. typically at this time, only some of the fields in the Practice Detail block are modify in and no Evidence Profiles are available. * Bronze status/Awareness raised As briefly as any evidence becomes available (i. e. , an Evidence Profile has been linked to the Practice Record), the status is set to Bronze level.For users, the Bronze Level status indicates that the practice has been nominated by our experts and user communities, and received a preliminary check for applicability. * Silver status/Evaluation performed When a sufficient set of Evidence Profiles is available, the BPCh experts will fill in the Practice Summary block and the status is set to Silver Level. For users, the Silver Level status indicates that the practice has been selected as promising ample to commission experts in the area to summarize key information. exploiters can see at a glance what they should know. * Gold status/Continuously maintained When the summary has been further evaluated (i. e. , vetted) by experts from industry, academia, and government, the status is set to Gold Level. For users, the Gold Level status indicates that the practice has been by dint of a rigorous analysis by a committee of experts in the practice itself as well as by user represe ntatives. Information on Gold Level practices contains the best and widest-ranging experiences we can find. confine STATUS OF THE BPCHWe have been piloting BPCh processes and tools by seeding initial matter. At this point the BPCh contains 51 practices at all levels of maturity. Practices that have progressed to Gold Level are those, like inspection/technical review, which have a prospicient history of published industrial experience. Many practices of interest in the area of systems and software acquisition have few documented sources of evidence or experience. Therefore, we are testing assorted processes for eliciting information from the workforce.Based on the recommendations of our User consultatory Group, the following types of practices are currently our top-priority areas for additional subject field *Earned Value Management, * Risk Management, * Information Assurance, and * Spiral Development Process. We hope that visitors to the BPCh tool will try out the offered featur es for providing short stories about their own experience with practices in these (or any other) areas. We encourage you to provide feedback as to whether you agree ordisagree withthe existent experiences that have been entered, or thoughts on our BPCh tool in general.LESSONS LEARNED Based on our experience with the BPCh program and other knowledge attention projects, we can formulate some observations which make useful rules of thumb for good practices to build such systems. The BPCh program has been organized along three parallel (but interconnected) tracks, which reflects our first lesson learned. LESSON 1 PROCEED IN doubled DIRECTIONS SIMULTANEOUSLY Progress in expression a knowledge repository needs to proceed in quaternary dimensions simultaneously content collection, tool development, and outreach.Although there is often a temptation to view these as tasks that can be done sequentially (e. g. , first the tool will be built, thencepopulated, and then it will be advertised t o users), we have found this to be an overlysimplisticview that diminishes the prospect of project success. Constructing the tool prior to collecting actual content and getting users feedback almost ensures that important user needs will be discovered late and will require much more travail to implement. Populating the content without getting user feedback leads to a high likelihood that the content will not unfeignedly address user needs.More importantly, content needs to come from the user biotic club, if the repository is to have a long-term life. We have found that for the research team to generate substantial amounts of content is a time-consuming way of recreating what many users already have at theirfinger tips. Finally, engaging in outreach and building excitement in the community of potential users runs the risk of all prototyping efforts When told how anything is possible in the final system, users often come up with many wish list features that are not really linked to their everyday needs.Moreover, users often getfrustratedwith the slow pace of progress when the system actually has to be implemented, and lose interest in the first place the system is fielded. To avoid these problems, we have adopted anincrementalapproach, with content and tool development going on simultaneously and outreach activities to the user community (such as booths at major conferences, or specific User Advisory Group meetings) planned at major milestones.Although this sometimes stretches resources a bit thin, we feel this approach has enabled us to lock up day-after-dayally with the user community, show them progress since the lastiteration, and get feedback on ever more mature versions of the system, with an initial body of content. LESSON 2 MAINTAIN A CONTINUOUS STREAM OF FUNDING Because of the interconnected nature of all the tasks listed above, having a stable funding stream is crucial.Requiring the team to take a rest periodfrom the project after a release is delivered leads to lost opportunities for user involvement (users find it hard to match their schedule to the development teams), leads to new content ideas that miss getting followed up on, may result in the loss of expertise if experienced personnel resources are in transition to other projects during the hiatus, increases the personnel learning curve encountered at restarts, and may result in flagging interest in the user community since momentum generated during outreach is lost.LESSON 3 RECOGNIZE THE RELATIVE MERITS OF CONTENT Our most important lesson learned is a direct implication of the BPCh vision There is no such thing as a Best Practice. Or, to state it more diplomatically No practice will be best for every project. Practices that are absolutely necessary for large, mission-critical projects may be too heavyweight for rapid prototyping or Web application development. The implications of this lesson are many.Perhaps the most important is related to the tone of the reco mmendations that users find Rather than arguing as an expert that readers should be following a given practice, or else they are doing something wrong, practices should be recommended to readers on the basis that projects of certain type(s) have found it useful. That is, rather than presenting aforegone conclusionto users, the system should aim at respecting users intelligence enough to enable them to draw their own conclusion, providing sufficient evidence as necessary for those decisions to be sound ones.LESSON 4 UNDERSTAND THE LIFE CYCLE OF BEST PRACTICES Practices (and practice information) are not static and have a real life round of golf. Majorparadigm shiftsin the software development world can have an impact on which practices are recommended. The practices that seemed to be good fits for most projects, when awaterfalllife cycle was the most common approach to software development, are not all equally applicable at the current time, wheniterative, spiral, and even agile app roaches are probably more representative of the state-of-the-art practice.Our recommendations regarding a incorporated life cycle for practice information are 1. A knowledge repository needs to be continually evolving by accepting information on topics of interest and making it available to users as soon as possible. While some quality checking is necessary to make sure that incorrect, misleading, or incomplete information is disseminated outward, it is better to get information to users as it comes in, than to wait and try to create something perfect.Users should be able to see a timestamp on all information so that they can see if the experiences related are impertinent and up to date or come from years ago. 2. However, the desire to get information out riotously should not interfere with the need for validation activities that provide higher confidence in the information. These additional levels of maturity should be noted, to give users more confidence in the information the y find, but should not be used as apreconditionfor displaying content. 3.Content needs to be retired when appropriate. Practices may have a natural lifespan, since the acquisition and development worlds relate to evolve and change on their own. Practices that were good 10 years ago may not be appropriate given todays constraints or technologies. To avoid users finding obsolete information in the repository, reports need to be generated periodically of which practices have received no updates or new experiences in the longest time. LESSON 5 APPLY AGILE STRATEGIES AND PROTOTYPINGTo create the front end of the BPCh tool, which helps users find candidate practices, explore possibilities, and get more information on practices of real interest, we have found that prototyping and agile strategies are extremely valuable for developing knowledge-management systems. on the button because of the need for parallel activities in different tracks, and the number ofstakeholdersinvolved (tool dev elopers, content gathering team, end user representatives, sponsor representatives), an agile approach is extremely valuable.The implementation of the prototype BPCh tool was carried on in two-week increments, at the end of which a releasable version was always available. At the end of each two-week period, a demonstration and planning meeting was held with as many of the stakeholders as could be present. This approach was necessary to help us ordain andprioritizethe evolving expectations of the users as well as the necessary changes that were suggested by the content development team, based on what they were finding. As part of this meeting we learned the following lesson LESSON 6 USE APPROPRIATE LANGUAGESpeak to the users in their language. Do not expect them to learn yours. We realized early on that having the greatest possible content in the BPCh repository would not be of much help if the users cannot find it. To address this we needed to provide multiple paths to the informa tion, so that users could select the path that made the most sense to them. Some specific lessons learned here included 1. Organize around common tasks. The best way to reach users is to organize the contents of the repositoryaccording toeveryday activities that the user performs.This helps users see the repository less as an additional activity that they need to make time for, and more as a value-added to the activities that already consume their time. In the case of BPCh, we added several such perspectives (i. e. , indexes to the content) based around activities of importance to different segments of the user community (e. g. , addressingCMMIpractice areas, constructing a systems design strategy, and referencing back to common guidebooks). 2. Push as well as pull information.Rather than always expecting users to take time to come to browse the BPCh tool, information can be pushed outward to the user on a periodic basis. For example, the user could select some practices of special interest, and when new experiences come in related to these practices a notification is sent via e-mail. 3. Match users to practices based on context similarity. Since no practice will be best for every project, it is important to match users to practices using context characteristics. This provides the users with a pick list of practices that may be useful in their particular situation, in ddition, it may alert the user to practices that they might not have known about previously. For example, if the user selects a few context variables that describe his/her context, then practices can be prioritized and displayed according to whether they have associated evidence provided by users with similar context information. This is a way of indicating that, even if the practice does not answer a specific search query, users like the current one have found this practice useful and it may be something the user should know. LESSON 7 DEMONSTRATE PRACTICAL EXAMPLES TO INTENDED USERTo engage in effective outreach activities, aimed at building up an interested and active community of users of the BPCh, we find the following lesson of relevance You can not show initial users an emptydepository. In line with the idea that building a tool like the BPCh needs to proceed on three tracks in parallel (front-end, content, and outreach) is the lesson that populating the content cannot come after the repository is built. Showing users a fancy front-end without an initial set of real content may get their interest for a short time period, but is not an effective way of building an active user community.Users need to see a small but representative set of content which they can respond to and start generating ideas for the next content or tool release. LESSON 8 modify CONTENT AND FUNCTIONALITY CONTINUOUSLY To keep interest engaged, when users do check back to the site they need to see that updates have been made since last time. Content needs to be continuously updated andrefreshedto stay informed of trends. If users ever become convinced that the repository does not get updated on a regular basis, this often spells the end of their involvement.Rather, they need to be motivated to come back often enough to find new things and hopefully, as they progress, be motivated to submit responses and ideas of their own showing emerging trends and keeping the content relevant. Thus, user involvement tends to build more user involvement. As users become interested enough to post comments or sendnew ideasto the repository, other users will continue to be interested to show up to see which comments have been added since the last time and possibly find something of interest to their current situationand more believably to find something applicable.One way we have experimented withto reinforce this conceptis to list on the front page of the BPCh tool the most recently added practices and highlight ones that have been promoted to various maturity levels (Bronze, Silver, or Gol d). Thus, one of thefirst thingsusers see is an indicator of how much progress has occurred since their last visit. CONCLUSIONS This article has presented some of the lessons learned with the BPCh program, which aims to document practices and quickly disseminate them to the users. The BPCh, which is based on the EMPEROR approach, makes use of a two-dimensional rating scale.These scales provide users with a quick overview of the trustability and maturity of the stored practice records. The scales allow users to understand and to draw their own conclusions based on a set of evidence from different contexts, from research studies as well as industrial experiences, and using measures at different levels ofrigor. Practitioners can rely on this information without reading in detail through the different evidence sources, unless they are interested in the very detailed level of information.In addition, ways to collect user feedback and inductance discussions are offered to allow a vivid a nd growing user community. While initial feedback regarding the BPCh tool has been supreme (Turner & Shull, 2005), we are continuing to improve the BPCh program and its associated tool through ongoing research, advisory groups, and user community feedback. We are interested in addressing such questions as How much extra effort tocertifyevidence sets and summaries as correct is worthwhile to users? or Are there subsets or types of evidence that users will find especially worthwhile? We invite you to take a look at our BPCh tool, available at http//bpch. dau. mil. We prize all feedback, whether it be submitted through the tool or directly to the authors e-mail. ACKNOWLEDGMENTS This research was supported with funding from the U. S. plane section of Defense (DoD), the theatrical role of the Secretary of Defense(OSD), and the Defense Acquisition University (DAU). We wish to thank the members of the BPCh team, from DAU, FC-MD, CSC, and SSCI, for the many copious discussions that hav e improved this work. REFERENCES Basili, V.R, Caldiera, G. , & Rombach, H. D. (1994). Experience factory. In J. J. Marciniak (Ed. ),Encyclopediaof Software Engineering (Vol. 1, pp. 469-476). New YorkJohn Wiley& Sons, Inc. Basili, V. , Zelkowitz, M. , McGarry, E, Page, J. , Waligora, S. , & Pajerski, R. (1995). SELs software process improvement program. IEEE Software, 12(6), 83-87. Dangle, K. , Dwinnell, L. , Hickok, J. , Turner, R. (2005, May). Introducing the Department of Defense acquisition best practices clearinghouse. CrossTalk, 18(5), 4-5. Defense Acquisition University.Retrieved from http//bpch. dau. mil Delphi colour Paper. (2002). Taxonomy Content ClassificationMarket Milestone Report. Boston, MA Delphi Group. Koennecker, A. , Jeffery, R. , & Low, G. (2000, April). Implementing an experience factory based on existing organizational knowledge. In Proceedings of the 2000Australian Software Engineering Conference(pp. 28-29), Canberra, ACT, Australia. Mendonca, M. , Seaman, C . , Basili, V. R. , & Kim, Y. M. (2001, June). A prototype experience management system for a software consulting organization.In Proceedings of the 13thInternational Conference on Software Engineeringand Knowledge Engineering (SEKE). Ottawa, Canada. Schneider, K. , Schwinn, T. (2001, June). Maturing experience base concepts at DaimlerChrysler. Software Process-Improvement and Practice, 6(2), 85-96. Turner, R. , Shull, F. (2005, November). An empirical approach to best practice identification and selection The U. S. Department of Defense acquisition best practices clearinghouse. In Proceedings of the quaternary International Symposium on Empirical Software Engineering (ISESE2005)(pp. 33-140), Noosa Heads, Australia. Mr. Raimund L. Feldmann is the technical lead for Knowledge and Experience Management at the Fraunhofer Center for Experimental Software Engineering, MD (FC-MD). Before he joined FC-MD in 2004, Raimund participated in several technology transfer projects in Germany and was similarly involved in the development of the Virtual Software Engineering Competence Center (VSEK) portal, funded by the Department of Education and Research (bmb+f) of the German Federal Government, to offer up-to-date Software Engineering knowledge to subject matter experts. E-mail address emailprotected umd. edu) Mrs. Michele A. Shaw is a Scientist at the Fraunhofer Center for Experimental Software Engineering. Michele supports clients implementing process improvement, measurement, and experience factory concepts. She has over 25 years of experience in Information Technology including software and service development, project management, quality assurance, client care andsubcontractormanagement Ms Shaw holds a BS in Business fromUniversity of Baltimoreand a masters in applied behavioral sciencefromJohns HopkinsUniversity. (E-mail address emailprotected edu) Dr. Forrest Shull is a senior scientist at the Fraunhofer Center for Experimental Software Engineering, MD (FC-MD). He is project manager and member of technical staff for projects with clients that have included Fujitsu, Motorola, NASA, and the U. S. Department of Defense. He has also been lead researcher on grants from the National Science Foundation, Department of Energy, Air Force Research Labs, and NASAs Office of Safety and Mission Assurance. (E-mail address emailprotected umd. edu)