Go to the table of contents.
The structure of this chapter is outlined in such a way that the first two sections 3.1 and 3.2 presents a selected set of generally accepted approaches to research and validation of the research followed by section 3.3 that describe how these approaches have been applied in the research presented in this thesis.
Research in design and product development is challenging task for several reasons. Design science and design research methodology are still evolving disciplines. As a consequence, no single agreed upon approach dominates. The scientific approach to the research task is therefore influenced to a large extent by the academic and industrial settings in which the research work is done. Furthermore, the research work reported in this thesis has been carried out to a large extent within the industrial setting in an action research oriented fashion complemented by activities that can be labeled as small case studies. These small case studies were carried out both to evolve the proposed concepts presented in this thesis and to obtain evaluations of their validity in an industrial environment. A consequence of this approach is that the design process under study involves many factors that influence both the research work and the behavior during actual design and development. The research work involves many roles and skills among the participating people, organizational systems, communication and information, commercial and logistic aspects, methods and tools, etc. Some of these actors and chains of events are central to the research work performed, whereas others are more peripheral, although they exercise influence on the research work in different ways and on different levels. Many disciplines in science are relevant, such as, engineering science, management science, social science, and computer science. Based on this extensive domain of issues it is clear that the research presented in this thesis will only be able to address and use a small sub-set of the approaches available in the field of design research methodology. The frameworks that have influenced and been used in this research project are presented in this chapter.
Carrying out research work in the context of an industrial environment obviously implies that the experiences of the researcher in that particular organizational setting play an important role. The scientific set of frameworks, approaches, and method thus forms a basis for structured reflection upon these experiences. Through the application of a scientific framework the problematic issues at hand and approaches to providing solutions for the identified problems can be generated. The problems identified and the proposed solution approaches to these problems can – to a certain extent – be validated through experimentation in the organizational setting in an action research oriented and case study based approach. A framework that describes this approach is the basic learning cycle (the Lewinian Experiential Learning Model, see Figure 4) described by Kolb (1984). Concrete experience from a certain problem or situation starts the process. Observations and reflections that are made on the basis of the knowledge obtained and experience lead to the formation of new or improved concepts and generalizations. These new or improved concepts and generalizations – in turn – lead to ideas for a new approach. The new approach, its concepts, and its generalizations with the corresponding implications must be formulated and then tested in new situations through active experimentation. Through this testing, of course, new experience and knowledge follow that make the start of a new learning cycle.
Figure 4: A basic learning cycle (adapted from Kolb, 1984).
This framework of a basic cycle for knowledge elicitation can – as mentioned in the text above – be further enhanced and systematized in order to provide scientific rigor through the application of additional scientific frameworks, approaches, and methods at different phases along the basic learning cycle. A number of such frameworks and approaches are described in the following text.
Action research is concerned with processes and phenomena that would not have occurred without active intervention from a researcher or research team (Wallén, 1996). This statement is an accurate description of the situational setting within which the research work presented here has been performed. However, while the theoretical field of action research is oriented more towards research on social systems and behaviors, it is an important theoretical background framework for this research as well. The relevance derives primarily from the fact that the reported research work has been performed within an organizational setting during ongoing product development and in operational teams.
An important aspect of action research is that it is concerned with a process in real action. Action research is not a combination of first research and then an application of the research results. Rather the application itself is a way to carry out the research, including data collection and evaluation. This means that from a scientific perspective it – in most cases – becomes important to describe the research process itself. However, in the case of this research the action research framework is not a primary objective of the research but rather a background framework; the emphasis on the research interventions from a process point of view is less important. The objectives of this research are the proposed concepts and models and not the process within which they were conceived and created.
Action research is basically a learning process and the most important results are the experiences and new knowledge acquired among the participants. Reports, such as for instance this thesis, will not cover the whole research process and will, in that sense, be incomplete.
Action research is mainly used when the researcher influences and studies the course of events in an organization or in society in general. Intervention study is another term sometimes used. The result of action research does not always correspond to successful results of the implementation created through the intervention. Provided that the intervention has led to new knowledge and that the reasons for the less successful practical results have been analyzed, the results of the action research may still be important from a scientific point of view.
One of the key features of action research is that it is the action in itself that is used to obtain a certain kind of knowledge. A potentially negative effect of this, however, is that it is hard to distinguish between the research and action. On the other hand, action research may in some cases be necessary from a methodological point of view. This is because, unless a phenomenon is studied in its real context, the validity of the results obtained may be unclear. A reason for choosing an action research approach is that one wishes to be able to closely follow a certain course of events or process. The fact that practitioners themselves engage in the research and that researchers participate in the operative action process may lead to new forms of theoretical constructs. Instead of generally applicable cause-and-effect relations these relations can be governed by the current context (e.g. local theory).
The case study as a scientific methodological approach (Wallén, 1996) has some overlap with action research in the sense that an individual case is under study. However, a case study does not necessarily imply the active participation of the researcher. Even though changes are not introduced through the active intervention of the researcher, the mere fact that something is being researched may trigger thoughts and reflections among those involved that lead to an introduction of changes. The main benefit of case studies is that things are being studied under real circumstances. Through the case study knowledge about the existence of a phenomenon is acquired. However, not much can be stated about the general occurrence of the phenomenon or about the underlying prerequisites needed for the same phenomenon to occur somewhere else. Another aspect to note is that it is not sufficient to compare states before and after a change since the outcome may have been influenced by other factors than those intentionally studied and/or used. Therefore it may be important to also follow the actual course of events during the change process (similar to what is done in action research but without the participation of the researcher in the process).
The kind of research work reported on in this thesis is – as described above – primarily conducted within an industrial organization. This means that the research work deals with problems as they are perceived by the researcher and others within the organizational setting. These perceived problems are then – by the researcher – subjected to a more formalized and systematic scientific framework. In order to do this the problems and the solutions generated and proposed will have to be matched with existing pre-knowledge and theories available within the scientific community. According to Mörup (1993) the scientific procedure is based on the ideas behind critical rationalism and fallibilism, in which existing models and methods are improved to provide a better description of the empirical reality. This is done through literature studies, logical structuring, empirical observations, thought experiments, etc. This approach is illustrated by Jörgensen (1992) and shown in Figure 5.
Figure 5: A method for applied research in which attention is focused on the interplay between theory and practice (Jörgensen, 1992).
The research has its starting point in a practical problem base where real phenomena in industry and in literature are analyzed and diagnosed. In parallel the problem areas discovered are analyzed in the context of the theoretical basis, in which new hypothetical statements on the nature of the problem are formulated. To check their validity and their applicability the solutions and hypotheses are applied in either the operational reality or in examples and test cases. The results are subjected to evaluation and critique by professional practitioners (designers) and the research community.
Much effort in this research work is centered upon the understanding of the information transformation processes occurring during the kind of product development considered. This kind of product development deals with how to create the required product variety to the customers while making effective and efficient use of product platforms. Eventually, some parts of the proposed concepts resulting from this research are expected to be implemented as computer-based tools that can support the business as a whole as well as individual engineering designers and teams. On a conceptual framework level the necessary ingredients for achieving such a goal were presented by Duffy & Andreasen (1995). This framework is illustrated in Figure 6 and consists of a phenomenon model as the first step in the analysis of reality and its problems, needs, and potential solutions. From the understanding provided by the phenomenon model an information model can be defined. The information model – in turn – provides the foundation for the conception and development of an implemented computer model. The models generated are furthermore checked and validated against the previous models.
Figure 6: Design research modeling approach (Duffy & Andreasen, 1995).
In order to contribute to the evaluation and progress of design methodology it is intended that models build upon the reality of a design context. These models are then continually evolved to develop methods and tools to support design work. Phenomena models are primarily based upon observations and analysis of the reality of a design context and the current use of the methods and tools employed. Where appropriate, the phenomenon models can be developed in more detail as information models. These information models seek to label all relevant object types and their relations. Computer models are often used to store such product information. At each stage any model can be compared or evaluated against any previous model in order to enhance understanding. The path from left to right explains the research carried out where phenomenon models are gradually formalized by means of information and computer models (Mortensen, 1999). The path from right to left is verification and validation, by means of which the models are confronted with and evaluated based on reality, i.e. empirical observations.
Models can be used for and are useful for many purposes. Models can for example be used to facilitate common understanding, improve the understanding of a task, etc. As shown in Figure 6 they can be further developed into computer models that can be implemented in tools used to support product definition and analysis, simulation, and other purposes. The models created and defined in the research work reported in this thesis have been partially implemented using so called COTS (commercial off-the-shelf) software.
As mentioned in section 3.1.1 the field of engineering design research is a large field that is still evolving and that includes many possible perspectives and aspects. To guide and focus the research work it is therefore necessary to apply some guiding mechanisms. One such framework that has been used to guide the research work presented in this thesis is described here. The framework in Figure 7, proposed by Blessing et al. (1998), defines a research methodology starting with a conceptual framework where criteria that describe the success of the research work and features that influence the success are identified. According to the framework, the first step is to define the criteria to be addressed by the research. These criteria include both a scientific research perspective and an industrial perspective. These criteria could be research goals, objectives, constraints, success criteria, and measurable criteria. Industrial criteria could be to reduce lead-time in product development or to improve quality. However, such high level industrial criteria must be further refined (broken down) into a set of more focused criteria and influencing factors that can actually be addressed within the scope of the research project.
Figure 7: Engineering design research framework (Blessing et al., 1998).
An understanding of the subject under study is obtained on the basis of observation and analysis (description I). Using experience and assumptions an approach to deal with the problem under study can be defined (prescription). If the prescription defined is applied to the subject under study in some form of intervention the relevance of the first understanding (description I) and the effects of the applied approach (prescription) can be observed and analyzed (description II). Based on this new understanding the first description and the prescribed approaches can be evaluated and validated or improved.
Pedersen et al. (2000) present an extensive framework for verification and validation. The framework is called the validation square. The purpose of the validation square framework is to address the issues of verification and validation in engineering design research. The area of design methods in the field of engineering design is an area of engineering research that relies on subjective statements and on mathematical modeling, which makes validation problematic. The validation square framework addresses the validation of internal consistency as well as of external relevance for some particular instances in order to build confidence in its general usefulness with respect to a purpose (see Figure 8). Pedersen et al. (2000) associate the usefulness of a design method with whether the method provides design solutions ‘correctly’ (effectiveness) and whether it provides ‘correct’ design solutions (efficiency). Correct in this context are design solutions with acceptable operational performance that are designed and realized with less cost and/or in less time. The validation process, supported by the validation square framework, aims at evaluating the effectiveness and efficiency of the design method, based on qualitative and quantitative measures.
Figure 8: The validation square (Pedersen et al., 2000).
The four validity constructs in the validation square are a combination of theoretical and empirical aspects with structural and performance aspects of the research results. In the assessment of theoretical structural validity two issues are considered: (1) accepting the construct’s validity, and (2) accepting method consistency.
References to established and accepted literature are a way to build confidence and acceptance regarding the validity of the individual constructs that, taken together, constitute the design method to be validated. An assessment of the consistency of the design method can be achieved through an analysis of each step (construct) of the method. Each step (construct) can be described and analyzed in terms of (a) available and adequate input, (b) anticipated output from the step (construct) is likely to occur, and (c) anticipated output is an adequate input to another step (construct). Furthermore, through the assessment of the available and adequate input required, an evaluation of the availability of this input in reality can be achieved.
The empirical structural validity is concerned with (3) building acceptance for the relevance of the example problems used to demonstrate the design method. There are three issues to be addressed: (a) build confidence in the individual constructs selected through an assessment that they are being applied in a similar way as they were generally accepted for in the reference literature, (b) clarify that the example problems used to demonstrate the design method are relevant and represent the actual problem area for which the design method is intended, and (c) ensure that the data used in the example problems can support the conclusions drawn.
In the assessment of empirical performance validity the issues considered are (4) accepting the usefulness of the design method for some example problems, and (5) accepting that the usefulness is linked to applying the design method. The usefulness of the design method can be demonstrated through representative example problems by linking metrics for usefulness to measure the degree to which an articulated purpose has been achieved. These metrics should be considered from both an industrial perspective and a scientific perspective. It remains to be demonstrated that these achievements are linked to the application of the design method. This can be done through an assessment of the contributions of each step (construct) individually. Through an assessment with and without the construct a quantitative evaluation can be achieved. Furthermore a comparison with alternative existing design approaches (rival theories) should be made.
The theoretical performance validity concerns (6) accepting the usefulness of the design method beyond the example problems. Ultimately, every validation rests on a socially justifiable belief, that is, faith. The purpose of going through the validation square is to present circumstantial evidence to facilitate a leap of faith, i.e. to create belief in general usefulness of the design method with respect to the articulated purpose. Generalization (i.e. support for external validity) is accomplished through assessment of the theoretical propositions, which is achieved by connecting the example problems to analytical generalization and theory development.
The importance and criticality of evaluation of results is well recognized in the scientific community. This section will therefore provide some additional and complementing aspects to the validation square framework presented above in section 3.2.1.
Depending on the nature of the reality studied and the theories used, evaluation procedures are not generally applicable but need to be carefully selected (Svensson, 2003). Two important dimensions in evaluation are verification and validation. Verification deals with the truth or accuracy and the predictive power of theories, methods, and models, whereas validity deals with their relevance and meaningfulness (Warell, 2001). Olesen (1992) suggests that research results should be evaluated with respect to five aspects: internal logic; truth; acceptance; applicability; and novelty value. Internal logic evaluates the condition that the results are based on known and accepted theories, and that there is a connection between the starting point, the hypothesis, and the result. Truth refers to an assessment regarding the ability of the theoretical and practical result to be used in order to explain "real" phenomena. Acceptance is an evaluation of whether other researchers accept the proposed concepts and/or theories as well as of whether practitioners and professionals accept and are able to use the theories, concepts, methods, and tools based made available through the results of the research. Applicability evaluates whether the use of the theories, concepts, methods, and tools provides for an increased probability for success with repeated use. It will not necessarily lead to success every time, but over a period of time it will give better results than if not used. Last, novelty value is an assessment that the research results involve a presentation of new solutions or introduce new ways to look at a particular problem. If the research result satisfies all five criteria, then it is possible to view the results as valid with quite good accuracy.
Additional dimensions for evaluation of research results are provided by (Yin, 1994) who identifies the following dimensions: construct validity, internal validity, external validity, and reliability. Construct validity is about establishing correct operational measures for the studied concepts. A prerequisite for construct validity lies in knowing what data to gather, how they should be perceived from the perspective of the phenomena studied and how to represent them. Internal validity is mainly concerned with causal studies, where an investigator tries to find dependencies between events. A complementary view of internal validity is presented by Buur (1990) who distinguishes four aspects of logical verification: (1) consistency, (2) completeness, (3) consistency of the theory with other methods and theories, and (4) case studies and specific problems that shall be explained by the theory. External validity (Yin, 1994) is concerned with the problem of whether a study’s findings are possible to generalize to beyond the immediate case study. A complementing view on external validity is transferability, i.e. to consider to what degree results can be transferred to another context (Guba & Lincoln, 1989). Reliability (Yin, 1994) is concerned with the probability of repeating a case study and obtaining the same results.
New methods or tools that have been developed need to be tested and evaluated in order to ensure their validity. Verification and validation of design methods and tools is a difficult task. Identifying the effect of a method or tool is difficult, partly because it is very much dependent on the context in which it is applied. This context is different for every design process, because every design project is unique. Mörup (1993) states that direct verification of design tools and methods are only achieved through successful application to a practical design problem. Mörup (1993) refers to Buur (1990) who argues that this method is unrealistic due to the stochastic nature of the design process and the large number of influencing factors that make repetition of experiments virtually impossible. Buur (1990) suggests two approaches for verifying the validity of design theory: logical verification and verification by acceptance. Logical verification involves examination of consistency and completeness. Consistency means that there are no internal conflicts between individual elements of the theory. Completeness is an assessment that all relevant phenomena observed previously can be explained or rejected by the theory. Logical verification further includes assessment of whether established and successful methods are in agreement with the theory and whether cases and specific design problems can be explained by the theory. Verification by acceptance involves an assessment of whether the results presented are accepted in the scientific community. Furthermore, verification of acceptance includes the fact that models and methods derived from the results are accepted and used by experienced designers.
An important aspect to note, however, is that any acceptance criteria should be considered with care. A new theory, method, or approach may be appropriate and correct even though it may take some time for it to achieve acceptance. Several factors contribute to acceptance, and it may be other factors than the new findings that are responsible for a low measure of acceptance. It is, however, an important aspect to consider during verification and validation and one that provides valuable information on the results.
The approach followed in the research work presented in this thesis is based on the step-wise refinements described in section 3.1.7. In an action research oriented approach (see section 3.1.3) observations, experiences, and analysis were combined in teamwork to define a prescription for how to incorporate support for product variants in the core product description. An essential part of the work of analyzing the observations and experiences and the definition of the prescriptions was the development of different models (see section 3.1.6). Models were created to describe and increase the understanding of the phenomena occurring in the business environment, to describe the information used and the structure of this information, and to define the approach to computer implementation of the new concepts developed. On the basis of these models, the prescriptions created were implemented in practice using a commercial PDM system and put into operational use in a new vehicle development program. In the case study environment (see section 3.1.4) obtained through this implementation and through the use of the prescribed methods and tools, the underlying assumptions of the prescriptions were challenged and experiences from the application of the methods and tools were gained. Based on the observations and experiences from the operational use of the methods and tools, some new insights were made and the proposed concepts somewhat refined and improved. The sequence of appended papers reflects to some extent the progression in the research process. The case study oriented approach (see section 3.1.6) described above has been complemented with studies of literature and scientific publications. The combined picture of these two streams of the research work (see section 3.1.5) is presented in this thesis.
The important issues regarding verification and validation have been addressed during the research work by observing the important aspects presented in section 3.2. The appended papers have been subject to full reviews before acceptance. Through the published papers acceptance by other researchers and novelty value have been established (Olesen, 1992; Mörup 1993; Buur, 1990). Through the practical implementation of the core concepts in a new product description and release system used for new car development at Saab Automobile, internal logic, consistency, completeness, applicability, acceptance by experienced practitioners, and ability to solve real problems have been demonstrated (Olesen, 1992; Mörup 1993; Buur, 1990). The consistency of the proposed new concepts with existing methods and theories (Buur, 1990) has been demonstrated by the positioning of the proposed concepts in relation to existing pre-knowledge referred to in this thesis (see chapter 4) and through the integrated framework of function-means modeling and configurable components (see section 5.4). To some extent this integrated framework suggests some level of external validity (Yin, 1994), i.e. a possibility to generalize the proposed concepts beyond the immediate case study in this research work. Since all results presented rest on one case study, it has not been possible to address the issues regarding reliability (Yin, 1994).
Contributions from this research are illustrated using the framework of Hubka & Eder (1988) which is characterized by four aspects (see Figure 9): (1) statements about technical artifacts, (2) statements about the design activities, (3) prescriptive statements, and (4) descriptive statements. The statements on design activities can be sub-divided into three areas: (a) design methodology or know-how, (b) theory and use of technical means, and (c) theory of design processes. The statements on technical artifacts can be sub-divided into two areas: (d) knowledge (scientific, technological, experiential, societal, etc.), and (e) theories about technical artifacts (e.g. Theory of Technical Systems, defined by Hubka & Eder, 1988).
Figure 9: Mapping of the contributions from this research (based on Hubka & Eder, 1988).
The contributions presented in this thesis are based on descriptive theories of development of technical systems that have their roots in Axiomatic Design (Suh, 1990) and the Theory of Technical Systems (Hubka & Eder, 1988). The extensions described in this thesis have been achieved by subjecting these theories to the research questions stated in section 1.3.1. These research questions can be viewed as centered on the need to design for variability. Design for Variability is a way to express a set of challenges and methods to deal with the issues that arise in the development of variant rich, platform-based products. In platform-based products a high degree of re-use of components is an important objective. However, the need to tailor the products coming from the platform includes an equally important objective to meet the needs of different users / customers. This second objective is the source of requirements for uniqueness. Modularization is traditionally regarded as a viable approach for trying to find a near optimal balance between these two contradictory objectives. This research work propose that an additional focus on how to make the design solutions themselves more adaptable through systematic parameterization is a complementary approach that can improve this kind of development even further. The concept of design bandwidth is introduced as a term to be used for this kind of approach. Thus, the objective of this research is to contribute to the understanding of the implications of these contradictory objectives and to provide a method for product representation in the context of design for variability.
Go to table of contents.