Strategic design

Strategic design

1. 1. Aspects of educational design

   • 1. Technical
     • detailed process with which any designer is familiar. It is focused on the design of individual elements of the product (e.g. a teaching unit; a professional development module; an assessment task). Technical design is focused on the end users and their environment (students and the teacher in classrooms; teachers in professional development activities; the diverse students taking a test, and those who will score their responses).
   • 2. Tactical
     • Is focused on the overall internal structure of the product (e.g. a multi-year set of teaching materials; a year’s assessment; a professional development package). Typically it involves such things as:
       • 1. Specification of core design principles
       • 2. Selection of specific learning and performance goals
       • 3. Specifying sequences and cross- connections within the materials
   • 3. Strategic design
     • Overall structure of the product set and how it will relate to the user-system. It applies in different forms to most of the products and processes that educational designers tackle: curriculum specifications; assessment; teaching materials; professional development processes and materials; building system capacity in various ways. Typically strategic design involves not simply the end-users (e.g. teachers and their students) but all the key communities involved who will affect decisions on the framework within which the users work – school leadership; school system leadership; politicians; parents; and various other professions, such as assessment designers and researchers.
       • Include things like:
         • Identifying a specific opportunity for improvement; Selecting a set of improvement goals; Designing the overall structure of a set of tools that can forward them; Choosing or designing a model of change (whether, for example, comprehensive or more specific; one-step or gradual; curriculum-led, assessment-led, or professional development led) along with the phases, pacing and timing of implementation; Identifying the resources that are needed to do the job well (how much design effort, trialling, implementation support, and of what kinds), and the compromises that are acceptable; Recognizing and questioning constraints from the client’s grand strategy (generic performance goals; alignment; model of change; top-down v proposal driven); and Advising the client on the likely implications of their various decisions, including their likely unintended consequences and uncertainties – and suggesting changes.
   • 4. Design control
     • How are design decisions made, and by whom? While all members of a design team will contribute ideas and suggestions on all three aspects of design, it is worth identifying how choices are made among the huge range of possibilities that any design task affords. The obvious principle is to make the best use of the diverse design talent available in the team. This hierarchy of decision taking will influence the design and its impact.

2. 2. Common failures

   • 1. Assessment. Their focus is on the statistical properties of the test and the “fairness” of the procedures, with little attention to what aspects of performance are assessed[9]. Policy makers talk and behave as though tests are just “measurement”; they choose simple tests because they are cheap and, if pressed, argue that the results correlate with more valid and elaborate assessments.
     • The design of well-balanced assessment in a form that can be used for accountability purposes has been a solved problem for many years. There are working examples around the world of high-stakes timed examinations that show what can be done, and how it can enhance learning. They are not perfect but are vastly better balanced than most current tests.
   • 2. Standards. Many current models of national and state curriculum specifications (“standards” in what follows) in mathematics and science are examples of bad strategic design – they have the effect in practice opposite to that intended. They actually drive down standards of performance in the subject.
     • How might one design “standards” that set clear learning and performance goals without narrowing the curriculum? There have been various attempts at improving criteria to include strategic and tactical skills (often called processes) at different levels. There is a fundamental problem here too: the same strategies and tactics help people solve problems across the range of difficulty. Again, it is tasks, not processes, that have well-defined “levels” of difficulty.
   • 3. The inadequacy of professional development strategies. Well-defined PD designs; observation of the fidelity of their implementation; and detailed observational feedback on teacher classroom behaviour.
     • While general pedagogical principles are important in teaching, good teachers also show a wide spectrum of specific high-level skills and teaching strategies.

3. 3. Features of poor strategic design

   • Strategic design is about ensuring that the product interacts effectively with the system it aims to serve.
     • Underlying causes
       • 1. Unintended consequences are a universal feature of poor strategic design. Policy makers assume that their initiatives will achieve their goals without negative side effects. Governments and their educational advisers regularly deplore “teachers teaching to the test”.
       • 2 Faith in “expert” advice is an equally common feature of poor strategic design. Policy makers believe that, if they gather together “some of the best minds in the field”[17], the advice they receive will enable them to achieve their policy goals.
       • 3. Neglect of “gaming” Government initiatives are invariably introduced by presenting a set of admirable intentions with which few would quarrel. However, there is little awareness that the rhetoric will be much less influential than specific changes that promise to impact on those involved – “teeth” bite, while talk can and will be ignored..
   • 1. Underestimating the design and development challenge is an obvious corollary of the mismatch between intentions and outcomes that is shown in the usual failure to achieve the planned positive goals, as well as the unfortunate unintended consequences.
   • 2. Timescale mismatch The timescale of politics (a year or two) is shorter than that needed for the design and development of substantial innovations[19], let alone that of implementing real educational improvement (a decade or two)
   • 3. Imbalance between pressure and support When human beings are asked to do something new, they usually need tools, training and other forms of support to become proficient. For professionals this is particularly important when the change is profound, challenging their beliefs as well as their well-grooved habits of day-by-day practice
   • 4. Casual commissioning Underestimating design and development challenges leads governments and some other funders to underestimate the importance of the commissioning process. On the one hand, they try to specify in too much detail aspects of the design that need to be explored; on the other, often driven by wanting the product urgently, they fail to ensure that the emerging design meets the objectives they had set out
   • 5. Unrealistic pace of change Governments often want to be seen to be “solving the problem”, moving swiftly to the end goal – through the introduction of a new curriculum, for example. A one-step approach with a clear end point has advantages. However, the size of the step is limited by how much change those involved can absorb and implement with the support provided through tools and collegial support through networks.
   • 6. Political need “to be seen to be doing something” In this media-driven age, where outrage sells better than good news, politicians are constantly bombarded with ‘public’ demands to improve this or that, or to “make sure that this can never happen again”
   • Contributions of education professionals
     • Typical symptoms of the inadequacy of the input of educational advisers include:They never say that something can’t be done, They never say “We don’t know how to do it”, They never try to define the time and resources that it will take to design and develop the tools and processes needed to achieve the policy goals and They ignore system realities of the various kinds described above.
   • The methodology gap
     • A common feature of the examples in Section 2 is that systematic empirical development through trials, before implementation, would have revealed the sources of failure and might well have suggested improvements in the designs – the standard methodology of systematic development. two elements in the standard innovation cycle are now firmly established as part of government policy making. Using medical nomenclature, they are:
       • Diagnosis: insight-focused research, much of it government commissioned, regularly provides policy makers with diagnostic information on the strengths and weaknesses of current practice in many fields, including education.
       • Phase 3 trials: pilot field testing of treatments before implementation for evidence on outcomes is Government policy; however, because these strategic decisions are not seen as a design problem by those who make them, the purpose of those at every level who take part in these pilots is to show that the initiative “works”, rather than to learn how to improve it.
     • The key gap in the methodology is a research-based link between these two elements, namely: Design and development of initiatives using research-based methods. This is analogous to Phases 1 and 2 of the development of treatments in medicine[24] – the initial small scale Phase 1 explorations leading, in selected successful cases, to their careful systematic development in Phase 2.
       • Such research-based design and development involves, sequentially: Review of research, of craft-based knowledge, and of earlier innovations; Design, imaginatively exploring a broad range of design possibilities; Development through an iterative process of feedback from small-scale trials;
         • Piloting in representative circumstances is the final step before large-scale implementation. Its usual role is a summative validation of the initiative, rather than providing formative and developmental feedback. The prior phases of research-based development, too-often by-passed in education, are where the product is refined through rich and detailed feedback, its quality and robustness enhanced, and unintended side-effects discovered.

4. 4. Successful strategic design

   • 4A Nuffield A-level Physics
     • 1. The course was developed in collaboration
     • 2. A radically new type of A-level examination was developed, reflecting the innovative nature of the course. The unprecedented number of components ranged from a multiple-choice test through more extended examination tasks to a student report on an individual experimental project
     • 3. Since entry to university in England largely depends on the results of A-level examinations, negotiations ensured the wide acceptability of the new examination, in particular for university admissions. The government body charged with oversight of the examinations had also to agree
     • 4. The construction of the course was seen as a piece of engineering, a job to be done despite inadequate knowledge of how some of the basic components in the learning process work.
     • 5. Two years of trials in a group of schools provided vital feedback, not only for the detailed design, but for acceptability by teachers, and for getting the timing right. These trials did result in some big changes to the original plans.
   • 4B Connected Mathematics
     • 1. Inherent quality of the material All authors will say that this, the tactical and technical design, is the key to success. One would like this to be true, and the quality of CMP is widely acknowledged. On the other hand, the traditional curricula that have dominated the market, and still have a large share, succeed despite their only obvious virtue being familiarity to the customer and client groups[31]. “That’s the proper way to teach math, like when I was at school.”
     • 2. Quality of the design team The success of the Connected Mathematics curriculum, which is written for both teachers and students, reflects the diverse talents of the design team. The team consisted of authors, graduate students, graphics designers, teacher collaborators, researchers, and an advisory board made up of mathematicians, mathematics educators, teachers, administrators, and parents.In addition, consultants from the sciences, engineering, reading, English language learners, and special education provided valuable insights for specific aspects of the curriculum.
     • 3. Understanding and growing a “niche market” The authors have long been at the heart of the main organizations in US mathematics education, not only NCTM but NCSM, the smaller organization of “math supervisors” in school systems who strongly influence the choices of materials. This has given them a deep understanding of the needs and constraints perceived by these key constituencies. This has informed the design of CMP. Those who were looking for real change, long advocated within this community, found a workable curriculum that met their ambitions
     • 4. Specific support for meeting strategic challenges The project recognized the challenges that implementation presents and offered specific guidance and support.
     • 5. National evaluation Driven by the ”math wars” controversy, the Bush administration commissioned an evaluation of the available curricula[32]. An expert group (not, on this occasion, pre-selected to produce “the right result”) rated Connected Mathematics as exemplary
   • 4C VCE Mathematics
     • 1. Changing a high-stakes examination The research study described below (Barnes, Clarke and Stephens, 2000) shows how the change in the high-stakes examination led to corresponding changes in classroom practice throughout secondary school.
     • 2. A fine balance of ambition and realism The changes sought and achieved were very ambitious, reflecting a world-wide consensus on best practice in school mathematics in a form that was (and is) rarely found.
     • 3. Development from feedback The curriculum and assessment authority was prepared to fine-tune the design to make it work.
     • 4. A consensus for change in the system The changes in VCE were mandated by the system as a whole; this design team used the opportunity provided by a climate in which “the status quo was not acceptable”.
     • 5. Tests as a monitoring device Assessing and authenticating extended pieces of student work is always a challenge. Tests have been used in various ways as part of this process; the use here as a monitoring device was both original and effective in restoring public confidence.
   • 4D Testing Strategic Skills – “The Box Model”
     • 1. While a funding agency provided the opportunity, the design initiative came from a group with many years experience and recognized skill in the design of innovation, a deep understanding of the area they chose, and knowledge of the results of its long-term research agenda
     • 2. The team was aware of the strategic design challenge and addressed the needs and concerns of all the key constituencies centrally involved – successfully, at least for a while.
     • 3. Well-aligned changes in the system’s high-stakes assessment, when achievable, proved a powerful lever for increasing impact.
     • 4. Both an immediate “backlash” and a natural process of gradual erosion are to be expected; reflecting resistance to change, they need to be covered by persuasive arguments as well as evidence, and by ongoing “engines for improvement”.

5. 5. Principles for strategic design

   • Strategic design principles
     • 1. System awareness: Seek to understand the dynamics of the system you seek to improve, in all its interacting parts, and use it to guide the strategic design of the innovation.
     • 2. Realism: Study the system as it is, not as it is intended to be, and the forces that shape decisions and actions of all the key groups, from politicians, parents and the media to teachers and their students; don’t assume resources that have not been available without valid assurances that they will be.
     • 3. Targeting: Be clear and specific about improvement aims, and the groups of users you are designing for – development should reconcile the goals and outcomes for those groups.
     • 4. Alignment: Try to ensure that the set of tools and processes you develop form a coherent whole, in themselves and in interaction with the rest of the system – all the key players should be aware and “on board”.
     • 5. Robustness and flexibility: Since unexpected shocks to your plans are inevitable, try to design the set of tools and processes so that various elements can function independently in a range of contexts of use.
     • 6. Consensus building: Seek consensus on goals and entailments prior to design and throughout the development process – a profession that speaks with one voice has more influence on policy than one where diverse opinions reach policy makers. Consensus does not just happen; it often needs to be built through explicitly designed processes.
     • 7. Communication and marketing: Be aware that any large-scale impact of your work will be influenced by the public, guided by the media. Improve your communication skills with these groups, and your network of contacts.
     • 8. Space for excellence in tactical and technical design: Work to retain as much space as possible for the creative talents in your design team, and the systematic development that refines the products – good strategic design is worthless without them.
     • 9. We must educate our masters”: Seek to make policy makers, funders, and designers aware of the crucial role strategic design will play in the success of the enterprise in turning its goals into large-scale impact.
     • 10. Big challenges need big teams: The range of skills needed to carry through a design and development program, with high-quality in all its aspects, needs to be reflected in the design team – often, particularly for large scale developments, only a multidisciplinary team can understand and work with the various communities that will interact with the product.
   • Strategic design issues
     • How big a step? How ambitious should a change be?[38] For a given level of support, if the step is too large, few will take it without stumbling; if it is too small, why bother? If we are ambitious, can we define and support a pathway of progress for the key performers, particularly teachers, so the latter can gradually move to match our ambitions in their classrooms?
     • ‘Big bang’ v incremental change Should we seek to achieve our goals as a single major change (e.g. introducing a new curriculum, as in 4A and 4B, or incrementally, as a planned sequences of changes (as in 4D)? Small steps can be less expensive, more easily sold, and more digestible to users. A comprehensive reform is more conventional, and more satisfying to many, including politicians who like to “solve problems” (though other fields, like medicine and engineering, move incrementally). The trade-offs are fairly clear; the best choice less so – and system dependent.
     • Time scales How can we meet short term political thinking and achieve anything useful? We have noted the fundamental mismatch between the time-scale of politics and that of significant educational improvement. Politicians need to show results well before the next election. Education systems are built around professionals, skilled in aspects of their work through well-grooved practices; changes in those practices, particularly those requiring new skills, take time – for the system as a whole, typically a decade or more.
     • Standard slots v new opportunities Should we try to improve an existing entity (e.g., a school subject or an examination) or to get a new one accepted, as a replacement or as an alternative? This dilemma has faced many innovators. For example, statistics education, which has developed internationally as a problem-based subject built on interpreting real data, has long been unhappy with being seen as part of school mathematics (though that needs much more of the same approach!). This discomfort remains but attempts to get a separate subject slot in the timetable have had limited success. (It is available as an option in the UK for age 14 upwards.)

6. 6. Improving strategic design

   • Long term goals
     • Insight-focused analytic research on working systems is the best route to diagnosis of problems and their likely causes;
     • A long term agenda for improvement is complemented, as in other fields, by a regularly-reviewed sequence of steps along the way with sensible timescales;
     • Good engineering, integrating insights from prior research and development, design research, systematic development and evaluation in depth, will produce the most effective solutions;
     • Strategic design of their initiatives should be as professional as the tactical and technical design already (sometimes) are, using the same methodology;
     • and Much better evaluation of products and initiatives in education[41], covering in some depth both the various outcomes and the conditions under which they were achieved.
   • Medium term goals
     • High-stakes targets (i.e. tests) for (teachers and schools) can distort priorities, ensuring that the implemented curriculum in most classrooms is no better than what is tested. The good news (as in Section 4) is the substantial evidence that better tests can be an effective lever for improvement. What is achievable within the timescale required and resources available is: An empirical question that can only be reliably answered by imaginative design, systematic development and evaluation in some depth;
     • Usually much less than is desirable – or is promised by “experts” who are keen to please government but have no valid evidence for what they recommend; and Will require funding with at least a few–year timescale, involving competitive design groups and independent evaluation with agreed criteria and methodologies (c.f. NICE in health care). After so many failures from “obviously needed” reforms, there is political capital to be gained from a sensible research-based approach.
   • Short term actions
     • Identifying, as in section 4, examples of successful design, then studying the various aspects of their strategic design in some depth, and in comparison with parallel innovations where these are available;
     • Identifying, and specifying in some detail, alternative models of change, analyzing their key features and the expected cost-benefit analysis;
     • Refining and strengthening evidence of payoff from giving medium-term support to high-quality design teams with proven track records in well-defined areas;
     • and Developing effective channels for communication and influence on policy makers – one of ISDDE’s prime goals to which this journal is a contribution.