John Hattie: Invisible Learning, Visible Amateur Science


The picture above: Many schools today are far more friendly and advanced as schools 40 years ago. Yet, in Hattie’s meta-meta study, schools of different decades are thrown into the same basket to search for statistical effect sizes of ‘what works best’. Picture Credit: Bundeszentrale für politische Bildung.

Introduction: Which factors? Measured how? For which purpose?

Let’s start with a joke about statistics:

Three statisticians went out hunting and came across a large deer. The first statistician fired, but missed, by a meter to the left. The second statistician fired, but also missed, by a meter to the right. The third statistician didn’t fire, but shouted in triumph, “On the average, we got it!”

Averaging effect sizes assumed by a normal distribution can be tricky business. When it comes to people (or deer for that matter), the average only exists as a social construct, an assumed variable that demonstrates reliable efficacy for most members of a given population. In Hattie’s case, however, things get more muddled up as we are dealing with not only different hunters that appear at different times of the day, but hunters who never knew what they were actually hunting for and who, after they randomly killed an unknown animal anyway, by mere chance, concluded a posteriori that they had definitely not killed a deer.

If this sounds like a rather confusing story with loose ends at every stage (to the point of not making any sense), you are welcome to the work of John Hattie who is cited and celebrated, around the world, as a ground-breaking educational researcher. I have written this little script to have a quick online reference to counter Hattie’s omnipresence.

I would not go as far as to accuse Hattie of pseudoscience, as Pierre-Jérôme Bergeron did in his review of Hattie’s statistics, since Hattie does not believe in supernatural forces per se (but omnipotent effect-sizes nevertheless), but I do assess his work as unbelievably sloppy and amateurish science to the point of where he is misleading the public. Other valid comments, such as on Hattie’s flawed statistics, include reviews by Neil Brown or Robert Slavin. Scott Eacott (2017) wrote an illuminating piece on Hattie and the damaging influence of his work on Australian education. For Eacott, Hattie’s work promotes a one-size-fits-all, neo-Taylorist approach to school leadership.

When John Hattie published his book ‘Visible Learning’ in 2008, he intended to pragmatically find out ‘what works best in education’. His basic idea appeared logical: the more studies can be collected about the factors that foster success in learning, the closer science might get to the Holy Grail of education. Hattie had hoped to find universal variables that, once and for all, would scientifically indicate what works best for students to achieve academic success. There are, however, a number of critical conceptual issues that Hattie failed to address. These are listed in the following.

The conceptual issues, in a nutshell

1. Lack of a clearly defined, valid and reliable construct

Hattie never clearly defined the dependent variable, which is student achievement. Since most studies use grades, Hattie (probably) assumed that grades are the smallest common signifier and denominator of students success. Hattie never clarified his all-encompassing variable or how it was concluded. He never explained how the variable of student achievement was constructed (based on which empirical evidence?) and in which context. If he had employed proper scientific methodology, he would have needed to conduct an Exploratory Factor Analysis (EFA) and a Confirmatory Factor Analysis (CFA), leading up to proper Structural Equation Modeling (SEM). Hattie never did any of this. He never defined the dependent variable he actually pretends to measure and which his entire book rests upon. There is no deer to hunt. Hattie is following a wild guess. If Hattie had indeed bothered to clarify the main construct of his study, he would not have been able to throw all studies, regardless of origin, quality or composition, into the grinder of a meta-meta-analysis.

2. The lack of validity of meta-meta studies for educational contexts

In ‘Visible Learning’, studies from the 70s and 80s are mixed together with studies from the 2000s. Seriously… can we blindly assume that no progress has been made in schools around the world for the past 40 years? Hattie disregards factors of pedagogical development and culture, public policy changes, historical and socio-economic circumstances. With all due respect, to not take such factors into consideration qualifies as junk science.

Another good example of how blind (and rather naïve) quantitative data can lead to misleading analysis is the success of Vietnam in the PISA Studies. John Jerrim wrote a detailed analysis of the topic. The underlying argument says that quantitative analysis can lead not only to wrong but grossly flaky results if contextual factors are not taken into consideration. In the case of the skewed PISA ranking for Vietnam, the flaw was the lack of a proper definition of the sampling population. Hattie is guilty of the same mistake.

Let’s assume for a moment, that Hattie’s could be right by averaging effect sizes of ‘what works’ and extracting the gospel of universal factors for promoting student achievement. If the silver bullets were indeed available, policies would be drawn out in an instant to spend budgets on ‘what works’ while schools would be told to scale back of what doesn’t. So if you happen to work at a school that is not average, according to the almighty meta-meta study, your luck has just run out. As an example, Problem-Based Learning (PBL) rates very low in Hattie’s universe of effect sizes, whereby in Medical Education, it has empirically proven to be significantly superior to traditional approaches and has been, for decades, firmly established as a State-of-the-Art pedagogy (see Wang et al., 2016; Sayyah et al., 2017; Zhang et al., 2015).

3. The limitations of effect size to inform pedagogy

Let’s assume that we compare apples with apples and oranges with oranges, in which case we can argue with effect sizes based on the average, e.g., the average size of a typical apple or orange, grown on a particular orchard. The average gives us, well, just the average. It does neither tell us the size of the smallest, nor of the biggest apples or oranges. Averages do not tell, how and why, the best and the worst students perform as they do.

But what is the basis of effects? Did students have 20 minutes for a test or were they given a full hour? Had a test been based on mere rote-learning or was it based on problem-solving and developing creative strategies? Has a school project been conducted as an individual task or as a group project? Was a lesson designed more in an instructional or more in a constructivist manner, or as a mix of both? And if yes, was this mix adequate and efficient to foster student learning? Hattie does not bother with such trivialities and suggests a one-size-fits-all model.

For Hattie to claim that one educational practice is per se ‘good’ (= large effect size) or per se ‘bad’ (= small effect size) defies all foundations of pedagogy. Didactic context is not considered in his work. A validated psychological or motivational model is absent in ‘Visible Learning’ as well. In order to make sense of any effect size in the context of classrooms, the inclusion of pedagogical and didactic models is a condicio sine qua non. An effect without context does not make any sense.

Below: Effekt sizes according to Hattie

hattie effects

4. Hattie’s self-contradiction

In concluding his book, Hattie differentiates between surface knowledge, deep knowledge and conceptual knowledge, all of which are regarded as essential. In one of his meta-studies, Hattie concludes that 60-80% of all knowledge created by tests, evaluations etc., qualifies merely as surface knowledge. This means that only 20-40% of all knowledge taught by teachers, passes as deep- and conceptual knowledge. Besides making a fairly bold statement – if this was indeed true, then all of Hattie’s measures of effect sizes would be practically meaningless: Once teachers intend to achieve deep- and conceptual knowledge, they would change their teaching strategies based on Hattie’s recommendations. In this case, predictably, most effect sizes would change significantly (!). This dilemma reveals another major weakness of his research: Hattie uses data ‘as is’ (which he later identifies as predominantly inefficient teaching) but uses the concluded effect-sizes from the same data to formulate a hierarchy of best practices. Things cannot be good and bad at the same time, can they?

Hattie’s cult following should not distract from the fact, that Hattie has neither contributed to the development of modern pedagogy nor has he informed sensible school policy. If I was looking for an inspirational figure (or charismatic guru) with a superior grasp on the dilemma of educational systems, I would look for someone like Sir Ken Robinson, or would professionally study constructivist pedagogy. Not all of Hattie’s writing is misguided. To give a more balanced review, I would need to mention his sensible ideas on feedback etc., but then again, these topics have been researched far better by others.


Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning. London: Routledge.

Scott Eacott (2017): School leadership and the cult of the guru: the neoTaylorism of Hattie, School Leadership & Management, DOI: 10.1080/13632434.2017.1327428

Zhang Y, Zhou L, Liu X, Liu L, Wu Y, Zhao Z, et al. (2015) The Effectiveness of the Problem-Based Learning Teaching Model for Use in Introductory Chinese Undergraduate Medical Courses: A Systematic Review and Meta-Analysis. PLoS ONE 10(3): e0120884.

Teacher Professionalisation in Digital Education (1): The Pragmatist Guide to Multimedia Prerequisites


Image: Kibo Robotics


As one of my friends amusingly commented, teachers have little choice today but becoming multimedia experts ‘if they do not want to embarrass themselves in front of the kids’. Obviously, digital media have started to change classrooms forever as new government guidelines put pressure on teachers to comply with developing digital competencies for themselves and their students. Apart from a lack of experience as users of digital media themselves, many teachers (and seasoned school planners alike) face a bewildering plethora of gadgets, software, and standards on offer. The inevitable question arises how academic staff can develop logical strategies for media-based courses that intuitively make sense to students. Ideally, any decent media course should allow students to advance fluently from a novice to an intermediate level, to finally achieving expert mastery and competence.

The first thing to notice about digital media is that there is an inherent logic to mastering it. Logically means in the following, that one type of media (and its level of mastery) is the prerequisite for another. For example, storyboarding is a prerequisite for planning a video production or HTML5 is a prerequisite for creating interactive images, graphics or videos. No HTML5 or JavaScript, no interactivity. It also makes, e.g., not much sense to start in data visualization if one cannot properly compile data with a spreadsheet. No spreadsheet, no data input, no data visualization, and so on.

Based on two decades of experience in designing media courses, I have compiled a little map that can help teachers make informed decisions. Experienced media buffs might skip this script. However, for anybody else, I have added a description to each topic to make the dependencies between the different stages of mastering media more transparent. Specialist areas such as Computer Animation, Virtual- and Augmented Reality or Robotics are not reviewed in this article since we focus on the basic media knowledge for teaching and academic staff at K12 and university level. Robotics have started to play a vital part in childhood education, To keep things simple, robotics is not part of the article and I hope to cover this great topic some other time. For now, I hope the described map, the Pragmatist Guide to Multimedia Prerequisites (PDF) helps. You can also scroll down to open the PNG-Graphic File in high resolution.


Since images, graphics and their representation form the base of pretty much anything else in the media universe, mastering them stands at the beginning of almost any design foundation studies. This includes e.g., learning the basics about layout, image formats, editing, how to create visual hierarchies, how to use colour, grid systems, and contemporary layout for different kinds of target audiences and applications. For students, photography is a nice prerequisite for digital image processing as they can work with their original high-resolution material, rather than downloading images from the web, which is fairly dull. Vector graphics are useful in applications where scaling and avoiding pixelation matters, such as in STEM classes.

Since teachers need to explain phenomena, graphics are a key ingredient in almost any class. The illustration of this article was e.g., created in Powerpoint. There are free as well as commercial photo-and vector editors available, ranging from one-time payments for software to subscription models such as Adobe, from online to offline solutions. I can highly recommend the book ‘Multimedia Learning’ (Mayer, 2001), or Richard Meyer’s lectures on effective multimedia composition. This media-didactic knowledge, combined with image and graphic-production skills, form the backbone of teacher communication in the classroom.


Studies of the history of film, cinematography, journalism or experiences in theatre are, among others, great qualifiers to feel motivated in video production. The most essential skill, however, is storyboarding, which has two important aspects. One is to tell a story well, the other is, surprise, financial planning, time management, and cost control. Once broken down sequence by sequence and scene by scene, producers can estimate production time, plan for shooting at different locations, schedule performers and participants, estimate the need for titles, graphics, visual effects, and so on. Even for a humble production of a Youtube learning video, proper preparation is of the essence if it should outlast the current semester.

I would offer at least two different levels of video post-production. On this introductory level, importing files, cutting, pasting and being able to output video in different formats is all that is usually needed. Ideally, vide-post exercises should be accommodated by teaching basic editing skills, such as knowing how to avoid jump-edits, how to create a smooth-flowing video or how to frame shots to support the narration at hand. Advanced video-post includes skills such as keying and using greenscreen, motion-tracking, and the integration of motion graphics (animated graphics), 2D and 3D compositing, colour grading or composing multi-track audio to support the video.

Most teachers won’t have the time (or nerve) to produce beautifully polished videos, but they can still produce effortlessly video messages or create sensible videos within their means. You may look out for commercial products like Camtasia Studio, which also works with HTML5 or its free alternatives. Below: Da Vinci Resolve, a freely available high-end video and audio editor and Camtasia Studio, a very popular editor for producing e-learning content.




HTML is very useful once teachers and students publish their own Webpage or Blog. A useful side-effect of students running their own Blog is to learn how to publish information responsibly and how to navigate the web maturely. Creating custom-tailored HTML pages, edit the code to fine-tune online pages (such as alignment, implementing media with iframes or checking for broken links), belong to the standard skills of anybody who uses the web to publish content. Besides, blogging is a nice way to get into writing, reflecting and finding one’s personal voice. Cascading Style Sheets (CSS) allow keeping the formatting of complex documents and the layout of web pages consistent. CSS3, the latest version, allows for the design of rounded corners, shadows, gradients, transitions or animations, as well as new layouts like multi-columns, flexible box or grid layouts. Teachers should master at least the very basics, such as HTML Tables, in order to be able to troubleshoot simple mistakes and errors occurring in online media.

Those who like to get beyond the basics might look at CSS Bootstrap, an open-source front-end framework for designing websites and web applications. It contains HTML- and CSS-based design templates for typography, forms, buttons, navigation and other interface components, as well as optional JavaScript extensions, that enable interactive functionality. CSS Bootstrap is also important when designing for mobile devices. Realistically speaking, I see few teachers professionalising this far.


Blogs, such as the popular WordPress, belong to the class of Content Management Systems (CMS). Blogs are great to foster writing, reflection, discussions, the publication of research, projects and engaging with literature. More complex CMS such as Joomla or Typo3 are more suitable for commercial ventures and involve a steeper learning curve. Since all CMS feature an HTML Editor and CSS, it is useful to learn simple BLOG management simultaneously to get fast into blogging. Personally, I would not sequence both skillsets, as I believe that students should start responsible publishing, academic writing and creating their personal learning space as soon as possible. I recommend a two-level introduction: one for beginners so they can start blogging, and one for advanced students and teachers to develop their online presence professionally since online portfolios become our satellites in lifelong learning.

HTML5 and Javascript

HTML5 is where things become interactive. For example, images can have Rollovers, videos can offer menus, interactive quizzes, or multiple pathways. HTML5 tools are also a great tool for teachers to obtain feedback from students on completed mini-tasks. It makes sense to introduce HTML5 after having mastered image processing-, video- and graphics-production, so teachers can create original interactive content. Luckily, most tools using HTML5 do not require coding or scripting, so users can focus on content, media-didactic and media-pedagogical issues. As HTML5 is applied to pre-produced media, it has been labelled as an intermediate skill for teachers to master.

HTML5 and Javascript are also useful for visualizing data, as you can see in these helpful scripting examples. Below: How to create a chart with curved lines.



As we find educational apps like sand on the beach, a central media competence is the critical evaluation of learning apps, both for teachers and students. Questions are: Does this work for me or my group? What are the benefits, what are the limitations? Do benefits outweigh the limitations? It is a skill that I would practice early at the beginner level since we like to implement those apps that we deem useful for our online and interactive media platforms later on. Below a screenshot of the popular Math App Geogebra.



Visualising data is a key skill for many subjects at school and university. Creating dynamic, interactive PDFs is a likewise nice (perhaps not must-have) skill as well and is mentioned here to give an idea. Before teachers and students engage in data visualisation, they should have at least basic knowledge of statistics and how to use a spreadsheet. It is recommended to brush up on basic competencies in these areas and to create entry-level refresher courses for those who do not feel confident. It is better to create no data visualisation at all, rather than to create visualisations that may mislead or wrongly inform the public. With going online, comes responsibility. This is how a critical review of statistics, their interpretation and visualisation should complement technical competencies. Look out for Tableau Public and other data visualisation packages.


Since LMS represent another Pandora’s Box altogether, it may suffice to say that these complex CMS involve practically all other competencies and therefore stand at the top of the media competency pyramid. Teachers and school developers require experience in collaborative course planning, media integration and social design. Most teachers will use an LMS to publish their courses online, this is how prior knowledge of HTML5, CSS3, learning apps etc. on are of the essence. Few LMS accommodate Mastery Learning paths (personalized learning) out-of-the-box to support weaker students and to optimize programs. Administering LMS is a new full-time job in digital schools and universities.


From my experience, always go for those choices you can afford and those that allow a sustainable development of competencies. If you aim for a professional level, subscription models may be a convenient way to have access to the latest multimedia suites. If you specialize in only one area, you may only need specific programs in your field of expertise, and so on and so forth. If you teach students, think of how your students have access to affordable tools after they graduate. There is no point in training students to high levels of media competency using expensive software packages or costly licensing arrangements, if after their graduation finances get tight and their dream of participating in the digital economy falls apart.

Literature: Mayer, R. E. (2001). Multimedia learning. Cambridge: Cambridge University Press.

Click below to enlarge in new window

Multimedia Pragmatist Guide 7


The TPCK Model of Technological Education: Explaining Everything, Achieving Little


Digital media are becoming more common in classrooms around the world. The underlying pedagogy and design of curricula determine the benefits to students. Picture credit: Northeastern University


The TPCK (also known as TPACK) Model of Technological Pedagogical Content Knowledge, originally proposed by Misrah and Koehler (Koehler & Mishra, 2005, 2008, Misrah & Koehler, 2006), has taken the academic world by storm by proposing a pragmatic and systematic theoretical grounding to assist teachers integrating content knowledge, pedagogy and technology.

The model was developed by the authors during a five-year cooperation with American K-12 teachers and the faculty of Michigan State University. Its theoretical foundation is based on Shulman’s concept of pedagogical content knowledge, in short PCK (Shulman, 1986, 1987). At the heart of PCK stands the intuitive argument that in order for teachers being able to render content knowledge (CK) accessible to their students, it is a prerequisite for them to transform content knowledge in a pedagogically sensible manner (P), integrating both expert content and pedagogical knowledge. To their credit, Misrah and Koehler pointed out that professional organizations such as the National Science Teachers Association (NSTA, 1999) and National Council for the Accreditation of Teacher Education (NCATE, 2001) acknowledged the value of PCK for teacher preparation and teacher professional development.

Description of the TPCK Model

The authors gradually extended Shulman’s model by the component of technology (T), translating it into what is known today as the TPCK (or TPACK) model. How does this model work?


Graphic: A visualisation of TPACK

Using Shulman’s metaphor of three overlapping circular domains, the model concludes a total of seven key areas of knowledge. Beyond the technological, pedagogical- and content knowledge (the body of expert knowledge about a subject matter), four new areas come into play:

Technology Knowledge (TK) is defined by the authors as the teacher’s ability to use and apply digital technology in the classroom.

Technological Content Knowledge (TCK) relates to how technology shapes and influences the representation of content. Recent examples would be, e.g., the examination of phenomena by statistics, predictive analytics or computer simulations, opening new ways of exploring the world that was not available prior to the advent of digital technology.

Technological Pedagogical Knowledge (TPK) is defined by Misrah and Koehler as the pedagogical knowledge about the educational benefit of digital tools. The authors cite as examples digital class record systems, online grading, discussion boards and chat rooms.

Technological Pedagogical Content Knowledge (TPCK), finally, as the cross-section of all areas ”… is the basis of good teaching with technology and requires an understanding of the representation of concepts using technologies; pedagogical techniques that use technologies in constructive ways to teach content; knowledge of what makes concepts difficult or easy to learn and how technology can help redress some of the problems that students face; knowledge of students’ prior knowledge and theories of epistemology; and knowledge of how technologies can be used to build on existing knowledge and to develop new epistemologies or strengthen old ones.” (2006, p. 1029.)

A Critical Reading of the Original Script

The original script contains two segments. The first describes the TPACK model, as outlined above, while the second segment was named ‘APPLYING THE TPCK FRAMEWORK TO PEDAGOGY’. This section advocates what is typically described in the literature as design thinking (Brown & Katz, 2009; Lietka et al., 2017; Plattner et al., 2016; Rowe, 1998). Mishra and Koehler define their approach explicitly as ‘learning technology by design’ (p. 1034).

In this application part, the authors forward some valid points that are typically conceptualized in constructivist pedagogy. These arguments consist of notions such as

  1. Technology changes fast, which is why teachers need to be able to update their skills, classroom content and mode of delivery continuously. This argument also entails (contradicting TPACK) that learners cannot primarily rely on content that is easily outdated, advocating the development of process skills to make appropriate content available and to relate facts with changing ideas and complex concepts.
  2. Learning happens in situated contexts, which is why digital technology needs to support the age of learners and their developmental needs, the type of school or a variety of individual learning styles. In design thinking, this notion is complemented by taking a user- or human-centred approach. This entails that pedagogy is not arbitrary as well.
  3. Competencies need to be developed: A mere wish-list of competencies does not tell teachers on how to achieve them. TPACK promises a structured approach to guide implementation.

On face value, the TPACK model appears intuitive in stating technology, content and pedagogy as the main drivers of education. Similarities to theories such as Maslow’s Hierarchy of Needs or Bloom’s Taxonomy arise in so far, as an additive modular framework is proposed to explain and guide the complex dynamics of learning processes. TPACK seems to intuitively answer to all potential combinations of its components by starting with simplified assumptions.

A closer view, however, reveals that TPACK rests on a number of problematic assumptions:

  • Isolated domains: The first assumption is that technology, pedagogical- and content knowledge exists in isolation and can be understood by relating them via distinct overlapping circles as a representation of empirical reality. The second reading of this first assumption is that technological, content- and pedagogical knowledge exists as an a priori construct, each type of knowledge separated from the learning processes of involved actors. Shulman’s original categorical distinction between pedagogical and content knowledge was not further questioned by Misrah and Koehler. Citing McEvan & Bull (1991) and Segall (2004), Archambault and Barnett (2010) pointed out that ‘content, in the form of scholarship, cannot exist without pedagogy, and (…) explanations of concepts are inherently pedagogical in nature.’ (p. 1657). To this extent, Shulman’s framework appears to unnecessarily complicate the reciprocal relationship between content knowledge construction and the corresponding metacognitive reflection (pedagogical knowledge) that accommodates and frames it. Adding technology to the equation does not solve the initial lack of construct validity.
  • The dominance of content knowledge: The second assumption is that content knowledge (CK) represents one of the main components of digital- or media-education and should be regarded as one of the main educational concerns. It is not elaborated and justified why and how that is. Modern pedagogy is concerned about the acquisition of competencies, personal resources and process skills, but not content per se. Content is framed by overarching concepts and therefore cannot be separated from discoursive epistemological access.
  • A Model without actors and goals in mind: The third assumption is that the placing of abstracted thematic blocks (T, P and CK) is the most appropriate manner to plan learning and teaching activities with and about digital media. By excluding the actors of learning processes from the model (teachers, facilitators, students or the school administration), TPACK is not developed as a student-centred framework, which should be a grave concern for any educator.

The Design Thinking Conundrum

The TPACK application section focusses on design thinking which is, by definition and its human (learner)-centered approach, constructivist in nature. Typical issues of design thinking do not deal with content dissemination, but problem-solving, the development of innovative and novel solutions to challenges, the creation of new knowledge and ideas. Design thinking is about facilitating testable hypotheses as well as developing and improving ways of communication and group collaboration. There appears to be an internal contradiction and gap in the approach suggested by Misrah and Koehler; this is to build on a teacher-centered model based on content dissemination on one hand and to promote design thinking, which is based on active self-directed learning, on the other. There is neither a clear explanation given nor a discussion facilitated by this significantly different and incompatible choice of philosophies. The elephant standing in the room is the role of constructivist pedagogy. Since the TPACK model had been originally developed in collaboration with K-12 teachers, it is likely that teacher-centred pedagogy was assumed as the default model whereby design thinking was added on as a technological-compatible, convenient afterthought.

In a blog post, Punya Misrah answered when asked about Problem-based Learning (PBL) as follows: “There is a value to small group activities as there is to a lecture. Clearly if we are emphasizing higher order thinking skills – we need to move away from simple rote-learning tasks. It is just that TPACK per se does not lean one way or the other. It does not speak to the broader goals of education. Some of my more recent work has focussed on these broader goals – and in that there are many areas (if not all) that we will be in agreement. Finally, TPACK does not privilege content, technology or pedagogy. In fact we have argued (quite clearly) that there are situations where one or the other is the driver.”

Misrah states unequivocally that TPACK is not a goal-oriented model. Educators may find an issue with the argument of taking a neutral stand on pedagogy. If we seek to educate strong self-directed, lifelong learners that build in intrinsic motivation in order to thrive in a digitized and globalized society 4.0, then a merely instructional or a behavioristic approach is not an option. Given that the goal of education is the empowerment of learners of all ages, TPACK cannot have it both ways.

Mishra is also wrong assuming an arbitrary relationship between pedagogy and technology. Empirical research has demonstrated that teacher-centred instructors use technology in significantly different ways as compared to instructors willing to change form and content of curricula via digital media or colleagues who have embraced constructivist forms of pedagogy (Schaumburg, 2003). His mention of online discussions as an example of technology driving pedagogy is not correct insofar e.g., teachers advocating rote-learning would simply not bother using online chats to disrupt their practice. It is the pedagogical attitudes of teachers that determine the use of technology in the classroom, not vice versa. Technology is not neutral in this regard since technology either supports or distracts from facilitating learning processes and achieving learning outcomes. In an arbitrary framework that facilitates any system, there can be no definite criteria for the usefulness or quality of digital media.

Conclusion: TPACK’s Lack of Empirical Support and Scientific Usefulness

While some academics find in TPACK a universal model for curricula design, teachers in the field may find it cumbersome, misleading and confusing to sort out the finer details of TK, PK, CK, TPK, PCK, TCK and TPACK in order to progress with the design of learning units. In their empirical study comprising 596 online teachers across the United States, Archambault and Barnett (2010) could not verify pedagogy, content and technology as three separate domains. The only domain that distinguished itself from the others was technology.

As expected, teachers struggled with making clear inferences to the overlapping fields. The authors specified the following example: “Three online teachers were challenged with separating out specific issues of content and pedagogy. For example, Item d – “My ability to decide on the scope of concepts taught within my class” was interpreted by two teachers as belonging to the domain of pedagogical content rather than content alone. The same misinterpretation happened with Item b – “My ability to create materials that map to specific district/ state standards.” The participants saw this item as a part of pedagogy content (Archambault & Crippen, 2009). These examples, coupled with the results from the factor analysis, support the notion that TPACK creates additional boundaries along and already ambiguous lines drawn between pedagogy and content knowledge.(p. 1659)

TPACK, to state the obvious, is not a theory that can be empirically verified. There are no dependent or independent variables defined (since all components are regarded as equal with no clear causal or conditional relationships implied), there are no null-hypotheses to test and there are no evidence-based outcomes to predict. TPACK failed in initial empirical research distinguishing its seven mutually separated domains. Most worryingly, however, the model offers neither a goal-directed framework that is concerned with the empowerment of learners in the use of digital technology, nor is it concerned about the effects of digital socialization trajectories on society.

It still needs to be demonstrated that the positive effects attributed to TPACK are inherent effects of the model and not based on the critical reflections that are bound to come up when educators discuss the integration of digital media for the classroom.



Archambault, L.H. & Barnett J.H. (2010). Revisiting pedagogical content knowledge: Exploring the TPACK framework. Computers & Education, 55 (2010) 1656–1662

Brown, T., & Katz, B. (2009). Change by design: How design thinking can transform organizations and inspire innovation. New York, NY: HarperCollins Publishers.

Koehler, M., & Mishra, P. (2005). What happens when teachers design educational technology? The development of technological pedagogical content knowledge. Journal of Educational Computing Research, 32(2), 131–152.

Koehler, M., & Mishra, P. (2008). Introducing TPCK. In AACTE Committee on Innovation and Technology. (Ed.), Handbook of technological pedagogical content knowledge (TPCK). New York: Routledge.

Liedtka, J., Salzman, R., & Azer, D. (2017). Design thinking for the greater good. New York: Columbia University Press.

McEwan, H., & Bull, B. (1991). The pedagogic nature of subject matter knowledge. American Educational Research Journal, 28(2), 316–334.

Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: a framework for integrating technology in teacher knowledge. Teachers College Record, 108(6), 1017–1054.

National Council for Accreditation of Teacher Education. (2001). Professional standards for the accreditation of schools, colleges, and departments of education ([Electronic version]. Washington, DC: Author. Retrieved June 27, 2004, from

National Science Teachers Association. (1999). NSTA standards for science teacher preparation [Electronic version]. Arlington, VA: Author. Retrieved June 28, 2004, from

Plattner, H., Meinel, C., & Leifer, L. J. (2016). Design thinking research: Making design thinking foundational. Cham, Switzerland: Springer

Rowe, P. G. (1998). Design thinking. Cambridge, Mass: MIT Press.

Schaumburg, H. (2003). Konstruktivistischer Unterricht mit Laptops? Dissertation. Berlin: Freie Universität Berlin. Internet-Dokument: [5.6.2015]

Segall, A. (2004). Revisiting pedagogical content knowledge: the pedagogy of content/the content of pedagogy. Teaching and Teacher Education, 20(5), 489–504.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.

Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–22.



Why Solidarity? Notes on Social Justice


Our Natural Sense of Solidarity is Both Necessary and Compromised

Our natural sense of solidarity with others is situationally compromised, no matter how self-glorifying we perceive ourselves. While we generally feel great empathy for the suffering of others such as e.g., on TV as we see children ripped apart by bombs in war zones or scrapping for food on third-world garbage landfills, we also close our eyes with ease once we are asked to pay a price for solidarity.  Refugees should receive help by the UNHCR, we all agree, but better not knock at our doorsteps. We know very well about the difficulties of countries such as Italy or Greece to deal with the influx of refugees and the unbalanced demands of the Dublin treaty, but conveniently close our eyes at the remote prospect of having to share the burden. We opt for solidarity when it suits us and as long as it does not cause personal inconvenience. Our natural sense of solidarity, mirror neurons activated or not, appears only of limited use to solve issues of social justice. Natural perceptions of human solidarity are easily corruptible, especially by politics.

Three Principles of Social Justice

Generally, we are dealing with three principles of social justice: the principle of need, the principle of performance and the principle of participation.

The principle of need says that people should get what they need. Like in Maslow’s hierarchy of needs, people need to be provided food and shelter, comfort and respect. This is fair. From a Machiavellian perspective we could state that unless the poor are not comfortably poor, social unrest will knock on the door. Typically, we all agree that people should be provided with the basics of life, such as medical care and education. Social justice is served when people are provided what they need.

The principle of performance says that it is justice when an effort is rewarded. We commonly hear two variations of this principle. (1) On a negative attributional note, it says: Someone ambitious cannot be treated the same as someone lazy. Be it grades at school or salaries – those who put in an extra effort need to be rewarded while others even may get punished. (2) On a positive normative note, it says: People are not equal. Some are content to live simple, happy lives while others are ambitious to live more demanding lives that also involve higher responsibilities. A surgeon should be paid more than a taxi driver because, obviously, a surgeon carries greater responsibility and requires years and decades of qualification while a taxi driver does not. When I visited China in 1985, corrupt taxi drivers actually made more money than doctors at a hospital, so assumptions on the principle of performance should not be taken for granted.

The principle of participation says that justice is when all people can participate in society and nobody should be excluded. People from socially disadvantaged backgrounds should enjoy going to the movies, theatres, order a pizza, not be discriminated against etc. just like anyone else. Disadvantaged socioeconomic groups should not be worn down to the point where they stop voting, lose hope in a better future, limit their world to immediate physical needs to merely live in the here and now. Participation entails the empowerment to contribute to the whole of society and to play an active part in it.

Conflicting Principles – Now What?

It is easy to figure out that although we generally agree on the justification of these noble and sensible principles, they can conflict with one another. Let’s take capital gain for instance. If I work hard enough, I accumulate capital. This capital is earned by my blood, sweat and tears and I shall, therefore, enjoy the benefits. So goes the traditional libertarian narrative. Soberingly, stock markets do not know blood, sweat and tears. They know numbers, patterns and trends, investor sentiments and market dynamics, but not private perspectives. Once people have accumulated enough capital, such as by inheritance, they do not need to work anymore for anyone, at least not directly. The non-working rich contrasted against the working poor violate the principle of performance: It is against the principle of performance that people enjoy the benefit of a luxurious and comfortable life without effort and qualification. In dictatorial states, such as North Korea, a small corrupt elite manages a different system in which people are rather poor together (low needs satisfaction, high social inclusion cemented by ideology) while performance is measured by the loyalty to the regime.

Currently, the USA under Trump promotes a Rusian model favouring oligarchs, reminiscent of feudal societies of the Middle Ages. In this system, a caste of ultra-rich and powerful individuals and multinational corporations govern the rest of society. Democratic institutions, such as the judiciary or media, are hollowed out and remodelled to serve the reigning caste. Trump and Putin essentially share the same idea of society. Eventually, large parts of the population become socially and politically excluded (and alienated), they will not be able to satisfy basic needs and, in addition, experience how honest effort and hard work do not translate into a better life. People work hard but stay poor. The American Dream has come to its end. How does this compare to North Korea where a large part of the population prefers staying poor together, united under a God-like leader?

The cited examples share a common trait – they are based on inequality. This is not inequality in people being different, which by nature they are, but by attributed inequality (in this case the ‘entitled’ versus the ‘non-entitled’ members of society). Inequality corrupts all three principles of social justice. In a society where rich and powerful elites reign there can neither be a fair gain by effort (since wealth and power are inherited and stays within the caste) nor can there be a fair social participation (little or no social mobility, Wallmart for the poor and luxury brands for the rich) nor can there be a fair distribution of social goods such as medical services, housing, a decent infrastructure, democratic political influence and education. Perversely, it is even beneficial to reigning elites if the rest of society stays poor, uneducated and easily controllable. Recent technological advances in Big Data and AI provide dominating classes with powerful tools to predict social motivation and to influence groups’ decision-making.

Why Solidarity?

Kant would have argued that since we are agents of reason, we cannot use others as a means to an end. Applying the Categorical Imperative, it would be immoral to instrumentalise others, or groups of others, for one’s personal benefit and gain since we would violate their autonomy as well as betray our own. But why should powerful elites care about Kantian reason?

The Habermasian argument goes deeper. Reasoning, unlike a Kantian metaphysical concept, renders for Habermas as a function of intersubjectivity and underlies social action. No matter how unequal and suppressive a regime might be, it can never fully silence questions of legitimisation and justification. Which good argument (that is not ideological and arbitrary) could legitimize inequality? Stakeholders in society can raise the questions of a fair mediation of applied principles at any time. Emancipating inquiry, suppressed or endorsed, is in principle always open and possible. Only in a society that is based on solidarity, claims to social justice can be mediated amicably, peacefully and productively. The process of globalisation supports Habermas’ view since questions about the legitimisation of systems and the distribution of wealth are amplified in an interconnected world – or ‘spaces of flow’, as Manuel Castells put it.

Beyond demonstrating the logic of solidarity via a communication-bound rationality, I could think of another good argument. This is that in a diverse, open pluralistic society any public solution-development tends to be superior and more sustainable as compared to single-minded propositions by dominating groups. Social coherence, shared wealth, democratic innovation and a future-oriented design of institutions are best mitigated within conditions of solidarity where stronger members openly support and foster the disadvantaged of society. We may in the short term suppress the perspectives of others, but only at the cost of negative long-term ramifications.

The more perspectives are invited to a discourse, the more differentiated and fair policies can be designed. Let’s call it the cultural problem-solving argument. It asks what kind of society we want to live in and which benefits, or suffering, we are we willing to impose on our members. In economics, solidarity is realized by a social market economy. It aims at empowering, not alimenting, the less well-off members of society.

Conclusion: Social Trust versus Living in Fear and Paranoia

Although principles of social justice compete with one another, we cannot escape the social and cultural conditions that we create. Solidarity, as an underlying expression of unity and framework to mediate claims to social justice, is a prerequisite for unbiased public discourse.  Solidarity goes beyond the weighing of interests or the balancing of influence since it aims at a cooperative system design. Cooperative design creates, beyond more sustainable solutions, the precious outcome of social trust. Correspondent Jan-Philipp Sendker illustrated the argument by telling the story of a Chinese billionaire (Talkshow with Markus Lenz on German TV, ZDF, on 30. November 2017).

Despite his incredible wealth, this man lived in constant fear of authorities and even of his family who, according to him, could not be trusted with taking care of his autistic son after he and his wife would die. This is how he asked Jan-Philipp Sendker if he could assist him applying for German citizenship. He had heard in the media that Switzerland, New Zealand and Germany were among the few countries where handicapped children could be institutionalized and trusted but not Chinese institutions. Not all the money and power in the world was able to provide him with the comfort of public trust.

Solidarity among all groups of society is a prerequisite for social justice. This is how divisive and polarizing populists run down countries by turning them into plutocratic oligarchies. It is not a trajectory that open Western societies want to follow.

Learning Outcomes for the 21st Century: How and What to Learn in an Increasingly Dynamic World

cat robot Toshifumi Kitamura AFP

Picture credit: Toshifumi Katamura 

This script as PDF: Learning Outcomes for the 21st Century, Kompa 2017

Which are the Necessary Conditions for Learning in a Dynamic World?

As a saying goes, ‘One man’s jungle is another’s rainforest’. The choice of educational outcomes relates to rather diverse socio-cultural, economic-political, psychological and philosophical assumptions so that we may never find a final, all-encompassing consensus on contemporary educational goals. Still, in order to derive at a sensible pragmatic result, we can ask critically for the necessary conditions that are required for people of the 21st century to constitute their lifeworld and systems, to use the terms of Jürgen Habermas. At the end of the day, our influence on the real world is the final measure of success. If only the less capable and competent run our world it demonstrates that our educational and political systems are at peril.

We know that epistemological competencies of knowledge construction are equally as important as the ability to communicate knowledge within society or to create new knowledge in context. To add to the list of expectations, we are aware that the challenges of globalized and digitized societies raise the bar for individual self-regulation. This means that people need to be able to cope psychologically with ongoing changes (such as how workplace changes affect one’s personal life), unlike traditional societies that are still based on rigidly-structured and predictable cycles of knowledge acquisition. A good example of this change is the new ideal of lifelong learning as well as the awareness of the increasing diversity and discontinuity of contemporary careers and jobs markets. On a societal level, things do not become less complex.

This is how, looking for relevant goals, it is not only important to secure better individual and social learning opportunities for young people but to empower them to develop, manage and improve the social systems they live in. This notion entails the fostering of systemic competencies. If people do not want to become passive onlookers on their lives, they need to be able to mediate the disruptions and conflicts arising from technological and economic developments. This is how the superordinate educational meta-goals need to assist sustaining the continuous improvement of individual, social as well as systemic conditions. Since self-governance and cooperative problem-solving play a major role in our historical situation of globalized and technologically transformed societies, we find ourselves redirected to the values of autonomy and solidarity of the Age of Enlightenment.

The Overarching Structure of Modern Educational Goals

Traditional education systems rely heavily on the acquisition of individual-cognitive competencies (such as, e.g., traditional reading, writing, arithmetic etc.) which serve as a resource to draw upon for the rest of life. Society 4.0, in stark contrast, requires continuous professional development, the situational updating of social and intercultural skills as well as restructuring our psychological organisation to accommodate the complexity of multi-dimensional change. Once we become aware of the new societal conditions governing the 21st century, we can paint a fairly coherent picture of the critical conditions that are needed for sustaining successful biographical life projects within an open democratic society.

In this light, Self-Determination Theory (SDT) (Deci & Ryan, 2012; Ryan et al., 2012) differentiates between the human needs for social relations, competence and autonomy. The latter relies not only on individual factors such as motivation and basic knowledge but on accommodating social conditions to empower autonomy. Since the cognitive acquisition of competencies remains the central topic in empirical educational science, the need for competency development is hardly an issue of controversy. Likewise, it is generally agreed upon that the acquisition of competencies depends on both individual and social conditions such as access to education for all, adequate support, well-equipped schools and small class sizes.

The concept of embracing both theory and practice corresponds to the German ‘duales System’ (dual system) which promotes a dialectic relationship between hypothesis-generation and application, similar to the idea of a scholar-practitioner. We know things by creating them. Instead of talking about the acquisition of competencies, psychologist Carol Ryff uses the term ‚Environmental Mastery‘, which points beyond an abstracted, context-unrelated acquisition of skills and knowledge. Her 6-factor model (Ryff, 1989, 1995) also forwards the question how people make sense of their lives, how they can find happiness and psychological well-being. Unfortunately, well-being and happiness do not play a role in the educational models proposed by the OECD.

The Social Construction of Individual Meaning

  • Positive self-concept and the Art of Living (Ars Vivendi)

Life goals and resulting life tasks develop through authentic experiences. A prerequisite to translate personal experiences into future-oriented concepts is a positive and self-regulating Global Self that remains active during all stages of life. The goal of education is thus to empower people to assume positive self-regulation and not only functional problem-solving. We need to develop an art of living which is able to safeguard our psychological well-being. We need to learn how to make and keep ourselves happy (in a eudaimonic manner) which entails embracing wisdom, courage, humanity, justice, temperance, and transcendence (Peterson & Seligman, 2004). Human flourishing is what makes life worth living.

  • Future-oriented developmental perspectives versus alienation and social exclusion

Since life goals are mediated socially, political and socio-economic systems lose their legitimacy the moment personal and systemic interests and activities cannot be mediated reciprocally. Cutting the ties between lifeworld and system renders the system meaningless and epiphenomenal for all democratic participants. Hence, it must be a goal of education to foster mediation skills (systemic competencies) between individuals and democratic institutions in order to align individual life projects with institutionally guaranteed rights and benefits. Constructive political participation depends on properly-acquired socio-political competencies (e.g., how to mediate conflicts collaboratively and taking others’ perspectives into consideration), which renders in the light of emerging populism a strong argument.

  • Environmental adaptation

People find meaning in new and novel concepts of structuring their lives. In economy 4.0, more people work in teams and enjoy the benefits of a high division of labour but they are also confronted with problems that previous generations could never have anticipated (e.g., try explaining a Distributed Denial of Service Attack threatening the survival of a rural community to someone who had lived some decades ago). In highly dynamic economies such as of the OECD countries, young people are required to ‘learn how to learn’, as first conceptualised by Alexander von Humboldt. Self-motivation in solving problems, conceptual thinking skills and able to work in cooperation with others become essential skills to survive in job markets that currently polarize into higher and lower qualified jobs and thin out medium-qualified positions. Inevitably pressure mounts on education systems to formulate new educational goals that are based on understanding, designing and regulating processes rather than teaching static academic knowledge which is of only limited value in practice. Traditional school knowledge may not vanish completely, but it is currently reinterpreted conceptually (such as favouring mental operators, such as analysing, synthesising and evaluating, see Bloom’s Taxonomy, over factual knowledge) and has to prove itself in the context of transferability within interdisciplinary study paths.

Connecting points between researchers

In the following some remarks on the chosen authors to exemplify modern educational goals. Deanna Kuhn (Kuhn, 1991, 20056) defined competencies to formulate and discuss rational arguments, already present in the work of Barrows, from an epistemological perspective. Howard Barrows extended the rational construction of knowledge towards metacognitive reasoning (Barrows, 1992) which, in the meantime, has been further differentiated into individual and social metacognition (Briñol & DeMarree, 2012). As described by Barrows, problem-solving skills depend on a number of discursive-epistemological (hypothesis guided, relating facts to ideas) as well as social-communicative competencies (rational practice, open inquiry and collaborative deliberation). Albert Bandura, one of the most influential psychologists of our time, emphasises in his latest publications (Bandura, 2006, 2008) the necessity of rational, future-oriented self-directedness and self-efficacy to guard individual and collective perspectives of social action. All leading researchers agree on the rational foundation of knowledge construction.

Finally, my choice of including Claude Robert Cloninger is somehow ambiguous since I call the all-encompassing influence of genetically determined personality traits critically into question. Still, Cloninger identified in his Temperament and Character Inventory (TCI) (Cloninger, 1994) important personality traits that are critical to future-oriented learning. These are in particular the development of personal resources, the ability to take responsibility, the social acceptance of others, the ability to empathise, openness towards new experiences and unselfish behaviour. These are noble qualities, we could argue with Bandura (Bandura, 1977), that can also be learned socially and are not exclusively determined genetically.

In conclusion, most leading researchers connect individual and social competencies with abilities of truth finding, concept generation and meaningful social action (Frith, 2012) that integrate systemic perspectives. The arising key argument is that individual, social and systemic competencies relate to each other in a reciprocally-interactive manner. Traditional education, in stark contrast, has primarily only focussed on the acquisition of individual and cognitive competencies. Active learning philosophy has added social skills, systemic competencies and a more advanced psychological regulation to the list of essential educational goals.

From coarse-grained to fine-grained educational outcomes

In the following, I have mapped the discussed educational outcomes within a matrix as a working hypothesis. Besides the findings of leading researchers, we can verify necessary goals by a simple thought experiment. All we need to do is to imagine the consequences of missing objectives, e.g., what would happen if students cannot relate ideas to facts, or if they are unable to work together with others, what if they fail to communicate their concepts to the public and so on and so forth.

An educational goal can be regarded as critical and necessary if its absence leads to logical contradictions, self-negation or compromises higher mental and psychological functioning. Necessary educational goals do not exist a priori, but they evolve from intersubjective relations, which means that the absence or deterioration of objectives (higher educational standards) leads inevitably to social pathologies such as the emergence of aggression-reinforcing group polarisation, the development of rigid social hierarchies, elitist privileges or establishing the permanent exclusion of minority groups.

The concluded critical educational objectives are listed in the following PDF as ‘Extended Educational Outcomes’ (Click here: Extended Learning Outcomes, Kompa 2017). The associated 24 criteria are by themselves latent variables that require operationalisation within didactic contexts. To this extent, the EEO should not be regarded as a standardised ‘one-size-fits-all’ model, but an array of logical building blocks that allow for an almost infinite number of pedagogically useful models. It would be insightful to investigate how qualitative and quantitative data of these latent variables could be integrated so that user-generated data-sets for the optimisation and enrichment of learning processes can be utilized more appropriately.

We have just begun to envision the design of more creative, innovative and more holistic schools that encourage the human spirit to flourish, rather than to stifle it. For now, I like to put forward these extended outcomes as a proposal in order to empower young people being able to master our increasingly complex world. Compromising these standards and settling for any lesser would render a huge disservice to upcoming generations that have deserved better.


While traditional education favours the development of (a) individual cognitive competencies, modern education encompasses in addition (b) social skills, (c) systemic competencies and (d) a more complex internal psychological organisation to empower learners of all ages. Learning outcomes are not arbitrary but are based on real-world environmental demands. The proposed model matches an earlier concept advocated by UNESCO titled ‘Learning the Treasure Within‘ (1996) differentiating between Learning to Know, Learning to Do, Learning to Live Together and Learning to Be. As leading researchers agree on the importance of a rational foundation of knowledge creation, the question arises how knowledge construction and extended environmental demands can be woven into a next-generation pedagogy.



Bandura, A. (1977). Social learning theory. Englewood Cliffs, N.J: Prentice Hall.

Bandura, A. (2006). Toward a psychology of human agency. Perspectives on Psychological Science, 1, 164-180.

Bandura, A. (2008). Toward an agentic theory of the self. In H. Marsh, R. G. Craven, & D. M. McInerney (Eds.), Advances in Self Research, Vol. 3: Self-processes, learning, and enabling human potential (pp. 15-49). Charlotte, NC: Information Age Publishing.

Barrows, H. S. (1992). The tutorial process. Springfield, Ill: Southern Illinois University School of Medicine.

Briñol, P., & DeMarree, K. G. (2012). Social metacognition. New York, NY: Psychology Press.

Cloninger, C.R. (1994). The temperament and character inventory (TCI): A guide to its development and use. St. Louis, MO: Center for Psychobiology of Personality, Washington University.

Deci, E. L., & Ryan, R. M. (2012). Motivation, personality, and development within embedded social contexts: An overview of self-determination theory. In R. M. Ryan (Ed.), Oxford handbook of human motivation (pp. 85-107).

Frith, C.D. (2012). The role of metacognition in human social interactions. Philosophical Transactions Of The Royal Society B-Biological Sciences, 367(1599), 2213-2223. Oxford, UK: Oxford University Press. doi: 10.1093/oxfordhb/9780195399820.001.0001

Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press.

Kuhn, D. (2005). Education for thinking. Cambridge, Mass: Harvard University Press.

Peterson, C., & Seligman, M. E. P. (2004). Character strengths and virtues: A handbook and classification. New York: Oxford University Press/Washington, DC: American Psychological Association.

Ryan, R. M., Legate, N., Niemiec, C. P., & Deci, E. L. (2012). Beyond illusions and defense: Exploring the possibilities and limits of human autonomy and responsibility through self-determination theory. In P. R. Shaver & M. Mikulincer (Eds.), Meaning, mortality, and choice: The social psychology of existential concerns (pp. 215-233). Washington, WA: American Psychological Association. doi: 10.1037/13748-012

Ryff, C. D. (1989). “Happiness is everything, or is it? Explorations on the meaning of psychological well-being”. Journal of Personality and Social Psychology. 57: 1069–1081. doi:10.1037/0022-3514.57.6.1069

Ryff, C. D. & Keyes, C.M. (1995), The Structure of Psychological Well-Being Revisited, Journal of Personality and Social Psychology, 69 (4): 719–727

What the OECD Findings on Students’ Collaborative Problem-Solving Skills Tell Us … And What Not

PISA-CPS-Girls-vs-BoysCongratulations, Singapore!

As one of the pioneers that have championed Problem-Based learning (PBL) in Singapore, I was delighted to see Singapore on the No.1 spot in collaborative problem-solving when the OECD (OECD, 2017) presented its results on November 21, 2017. Many years of hard work by consultants like myself and government investment into a more student-centered pedagogy have, obviously, paid off.  Still, we need to be prudent on how to interpret the results since there is much more implied in the study than meets the eye.

Lesson No.1: Collaborate problem-solving is the exception to the rule, even among top performers

Graphs, like the one above, seldom tell us the overall picture. One remarkable key-finding of the study was  that (highlights by me) ‘(…) on average across OECD countries, not even one in ten students can handle problem-solving tasks that require them to maintain awareness of group dynamics, take initiative to overcome obstacles, and resolve disagreements and conflicts.’ (OECD, page 5). The study points out that even for top-performer Singapore only one in five students attain a high level among the cited criteria, while three-quarters of students are able to address problems of medium difficulty and can integrate diverse social perspectives. Collaboration as a key competence of the knowledge society (Moshman & Geil, 1998) appears rudimentary in practically all developed nations. The results reveal that there is much room for improvement across the board.

The unexplained gender gap

One of the central graphics and headline presented to the media by the OECD organisation (above) suggests that girls categorically outperform boys in collaborative problem-solving skills, which is not the case. Similarly, in a previous study, boys were found to outperform girls in individual problem-solving. Gender differences are statistically significant, but as in all statistics, this means that in reality there is still a large overlap between the better performing boys and the not so well performing girls (or vice versa, when looking at individual problem-solving skills). The authors of the study do no try to explain the international gender gap. They speculate that girls might simply be more receptive to interpreting nonverbal cues (Hall & Matsumoto, 2004; Rosip & Hall, 2004) since the gender gap cannot be explained sufficiently even after accounting for better reading literacy among girls.

Another reason might be found in different age-related competencies between boys and girls. Girls tend to mature faster than boys. This is how longitudinal analyses would be in a better position to explain underlying developmental factors. Judging from my experience with adolescent students, the gender gap diminishes as student populations grow older. In support of this hypothesis, the earlier maturation in girls has been associated with different neurological development (Lim et al., 2015). If varying neurological development could be identified to impact collaborative skills, the gender gap might not qualify as a solid predictor of collaborative skills in adulthood as data may suggest at first sight (see Figure V.4.4 below).

Looking at top-performers Singapore, Japan and Korea, the cultural influence on collaborative skills in interdependent Asian societies (Fiske et al., 1998) who also assign a high social value to education would be another worthwhile topic of investigation. As can be concluded from data, girls do slightly better than boys while some cultures do notably better than others. However, cultural differences clearly outweigh gender differences.

by gender

The big question: Is learning still enjoyable?

To facilitate lifelong learning, learning itself should be an enjoyable, motivating and insightful process. Learning should take place within a positive social environment and it needs to develop students’ personal resources. Although the significant effect of positive social relations for collaborative skills has been emphasised in the OECD study, there is no explicit connection drawn to problem-solving made in the classroom.

The generally stricter and more rigid learning environments in Singapore classrooms do not compare, by a wide stretch, to the more explorative and intrinsic motivation-based classrooms in Finland. This is how, to me, the psychological winner of the OECD study is Finland. Finland demonstrates that a nation can be a leader in collaborative problem-solving while advocating a student-centred, active learning pedagogy at the same time. This fact leads to another scientific blind spot, which is the issue of developing a sustainable intrinsic motivation to solve professional and personal problems throughout the lifetime. In the meantime, the successful alternative approach in Finland has been recognized in Singapore on a ministerial level (Sinnakruppan, 2017).

Lesson No.2: Problem-solvers are not necessarily innovators and entrepreneurs

With the promotion of collaborative problem-solving skills, Singapore had hoped to create an innovation hub reminiscent of an SE-Asian version of Silicon Valley. Although Singapore students fare well in problem-solving, innovation and entrepreneurship did not materialize to the extent it was anticipated by the government. Some factors inhibiting innovation appear to be the cultural habit of relying on a centralized administration, the unwillingness to take risks and to exchange ideas (Wan et al., 2005).

Although I am an ardent supporter of PBL myself, I had to learn over the years that problem-solving and entrepreneurship require different skillsets. Entrepreneurs display a high degree of frustration tolerance and are willing to take above-average risks. Entrepreneurs learn from failures, evolve advanced mental abilities to simulate future scenarios and develop high motivational levels in support of personal perseverance – all qualities that collaborative group processes do not necessarily imply. Innovators need to be brave: The truth is that more innovative ideas have also a higher probability of failure.


One of the key takeaways from the latest OECD study was that collaborative problem-solving is still in its infant stages, even among the top performers. Averages do not represent the stunning underdevelopment among practically all nations. We can agree with the authors of the OECD study that collaborating students only mature within collaborative schools. Beyond the mere measure of cognitive competencies, the development of personal resources and social skills seem to pave the way to succeed in the emerging knowledge societies.



Fiske, A. P., Kitayama, S., Markus, H. R., & Nisbett, R. E. (1998). The cultural matrix of social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), The handbook of social psychology (pp. 915-981). New York: McGraw-Hill.

Hall J.A. & Matsumoto D. (2004), Gender differences in judgments of multiple emotions from facial expressions, Emotion, Vol. 4/2, pp. 201-206,

Lim S., Han C.E., Uhlhaas P.J. & Kaiser M. (2015). Preferential Detachment During Human Brain Development: Age- and Sex-Specific Structural Connectivity in Diffusion Tensor Imaging (DTI) Data, Cerebral Cortex, Volume 25, Issue 6, 1 June 2015, Pages 1477–1489,

Moshman D. & Geil M. (1998), Collaborative reasoning: Evidence for collective rationality, Thinking and Reasoning, Vol. 4/3, 10. pp. 231-248,

OECD (2017), PISA 2015 Results (Volume V): Collaborative Problem Solving, OECD Publishing, Paris.

Rosip J.C. & Hall J.A. (2004). Knowledge of nonverbal cues, gender, and nonverbal decoding accuracy, Journal of Nonverbal Behaviour, Vol. 28/4, pp. 267-286,

Sinnakaruppan S. (Nov 26, 2017). Why Singapore’s education system needs an overhaul. In: Todayonline. Retrieved from:

Wan D., Ong C.H. & Lee F. (2005). Determinants of firm innovation in Singapore, In: Technovation, Volume 25, Issue 3, 2005, Pages 261-268, ISSN 0166-4972. Retrieved from:

About the Methodology of Social Change


The emergence of cooperative research

The Problem of Traditional Research Cycles

The biggest challenge to social sciences is the alienation between research and application. In social sciences and education, few empirical findings find their way back into improving everyday life and not everything that is being researched is of relevance in real-world settings. The gap between theory and application, between expert cultures and social actors, is grounded in traditional belief models.

Typically, it is assumed that a social problem, which is identified by researchers and decision makers, requires quantitative analysis to be objectively addressed. The tool of choice is empirical studies that reveal the causal, correlational and conditional relations of a social problem, e.g., by employing statistical methods such as regression models or structural equation models. At the bottom of the scientific hierarchy, we find smaller single-case studies, followed by correlational research, quasi-experimental studies and on the top meta-analyses and Randomized Controlled Field Trials (RCT/ RCFT). The latter is highly structured for the quality of reporting ‘enabling readers to understand a trial’s design, conduct, analysis and interpretation, and to assess the validity of its results.’ (CONSORT, 2010).

A problem that science despite all rigour, however, cannot reflect is the roles of involved decision-makers who interpret and apply scientific studies on behalf of their employees, staff as well as their users. A good example was the PISA studies that had a major political influence on educational governance across Europe. Local school principals, teachers, parents and students could only watch in disbelief how politicians surrendered to a merely economized view of education based on the measurement of few scant competencies. The top-down approach of monopolized research implies a number of problems when looking at developing sustainable social solutions. The most prominent critique is that social actors are degraded to mere data sources, for example via survey-based research, but they are not included as actively contributing rational agents.

This imposes four fundamental limitations to improving social systems:

(1) CONSTRUCT COMPLEXITY AND ENVIRONMENTAL VALIDITY: The complex nature of people’s lifeworld (their needs, interests, perspectives, motivations, ideas and wishes) is excluded in traditional research. The traditional objectifying approach, e.g., by limiting questions and potential answers of social actors to Likert Scales, imposes a reductionist perspective right at the beginning of the research process.

Traditional research implies a number of conceptual assumptions that may not coincide with the context in situ. Quantitative studies work well in settings where a problem can be clearly defined beforehand, such as medical problems with experimental- and control groups, but they necessarily fail in the case of social solutions development that depends on the active and adaptive contributions of social actors. In this case, the relation between research and research subjects is of an entirely different nature – one is descriptive while the other is participatory.

(2) DISEMPOWERMENT: This argument states that social actors are denied ownership of the problem at hand. By circumventing the perspectives of affected parties, participants of social transformation are excluded from the discourse to improve their environment. As a result, frustration and resentment may set in as participants are degraded to passive onlookers on the implementation of top-down policies. In addition, research and policy-making lose individual and collective meaning which otherwise would have emerged via active participation.

(3) SOCIAL INPUT AND DEVELOPMENT: Traditional empirical research rests on a number of unreflected assumptions. For example, which should be the desired effects of an intervention from the perspective of social actors, which noticeable criteria can measure these effects and which local resources are available to accommodate improvements? It is one perspective to abide by professional standards, such as promoting the acquisition of competencies or to develop individual autonomy, and another to empower social actors to work on solutions taking their unique cultural tools, personal motivation and local resources into account.

(4) SUSTAINABLE OUTCOMES: Social models that are created in cooperation naturally develop areas of corresponding responsibilities. Systems, where policies are imposed onto populations externally, suffer from the drawback that nobody feels in charge of managing them due to the lack of ownership. People assume responsibility for the systems they create.

The development of participatory models

We may take schools as an example. How a school deals with diversity, heterogeneity, its psychological climate, social support networks or mechanisms for continuous improvement is entirely based on the coordinated effort of the administration, teachers, students, their parents and the social context at large.

Due to the need for consensus, social problems and their consequences need to be set in relation qualitatively (Which is the meaning of problems to the individual and which are their social impacts ?) as well as quantitatively (How do problems scale in the social sphere?). Typically, such questions are more adequately addressed in mixed study designs. Mixed-study designs, unlike large quantitative studies, recognize the complexity of social transformations but still fall short of the criteria of democratic empowerment and cooperative local development. New innovative scientific approaches that are based on the input from social actors will be discussed in Part 2 of this essay.

From a data perspective

From a data perspective, we could state that in traditional research a fixed statistical method, based on collected data sets, is employed to determine the significance of effects or the causal interdependencies of factors within a construct. By contrast, in cooperative research, we need to design logical operators (methods) in such a way that we derive at a measurable, computable result. The traditional statistical method is fixed by its choice of initial study design, while cooperative efforts are based on the premise of performing adaptive process changes (competitive prototyping) in regards to achieving desired outcomes.