Thursday, 3 April 2014

9 things you should know about the new PISA "creative problem-solving" test

Today sees the launch of the first international test of "creative problem-solving". It is the latest addition to the suite of PISA tests run by the OECD which have become hugely influential in global education policy-making.

This test was taken by pupils in late 2012 at the same time as PISA tests in maths, science and reading but the results were held back for a separate launch. I was invited to a pre-embargo briefing yesterday and the information here is taken from a mix of the published reports and answers given by OECD experts at the briefing.

1. The purpose of the test was to measure students ability to solve problems which do not require technical knowledge. The PISA subject tests in maths, science and reading are also based around problem-solving but they do require knowledge in these subject areas (e.g. mathematical concepts and mental arithmetic). Examples of questions include working out which ticket to buy at a vending machine, given a list of constraints, or finding the most efficient place for three people to meet. Unlike the subject PISA tests it was completed on computers which allowed for more sophisticated interactive assessment.

2. Overall the results correlated fairly closely with the PISA subject tests. Unsurprisingly students who are good at maths problems are also good at ones involving general reasoning. The correlation with maths results was 0.8 and with reading was 0.75.

3. But England was one of the countries that did significantly better in this test than in the subject ones. It came 11th overall but the individual rankings are misleading. It makes more sense to think of clusters of countries that did about as well as each other. The leading group of seven consists entirely of Far East countries and jurisdictions. England is in the second group with countries that traditionally do well in PISA like Australia, Canada, Finland and Estonia. Then comes a third group made up other larger European countries and the United States. The countries below the OECD average are primarily smaller European countries and developing nations.

4. This is unhelpful for a number of the big narratives in English education policy. It undercuts the "England is falling behind in the world" narrative so beloved of right-wing newspapers. On a test of intellectual reasoning (which is what this is) our 15 year olds do as well as any other nation bar a small group of Far East jurisdictions (only two of which - Japan and Korea - are not cities or city states).

5. But it's also perhaps unhelpful for those who argue that our education system is dominated by an obsession with tests and narrow curriculum knowledge. It turns out we're actually pretty good at "21st century skills" already. Our students performed better in this test than you would expect based on their maths, science and reading ability. Likewise all the employers arguing that our system isn't delivering the kind of problem-solving skills they need should reflect on these results.

6. The reason England outperformed it's subject PISA scores is that students at the top end did better on the problem-solving test than on the subject ones. Students at the bottom end did no better. This suggests that we're doing something with our more gifted students that we're not doing with our weaker ones. In other countries - e.g. Japan - the opposite was true weaker students did better in problem-solving than subject tests but the strongest ones didn't.

7. In England there was no statistically significant gender difference in performance on this test (in maths and science boys do better; in reading girls do). Interestingly immigrants scored below non-immigrants which is a change from the maths and reading tests where there is no significant difference.

8. The domination of Far East countries puts pay to the notion that their success in PISA subject tests is somehow down to rote-learning or fact-cramming. It also puts pay to the idea that all Far East systems are the same. While Shanghai and Hong-Kong are still in the top group they did much worse on this test than would be expected given their stellar scores in maths, science and reading. Conversely Korea, Japan and Singapore all did better than would be expected.

9. While the test results are interesting they don't tell us why some countries do better than others. Both Singapore and Korea - who come top - have both tried over the past few years to add "21st century competencies" to their curricula to make them less purely focused on academic knowledge. But it's unclear whether their high scores in this test are due to that or because their traditional strength in the academic basics transfers to "creative problem-solving" tests of this type. The OECD presenters were clear that they thought it was impossible to teach problem-solving skills in the abstract without content, but they also felt it was possible to embed them in a knowledge-based curriculum.


  1. I am wondering why England chooses to participate (assuming that we do make this decision)? The data from international tests doesn't appear to help formulate effective national policy, but does create huge amounts of heat and sticks to beat people with.

  2. Very simply, we underperform our national IQ on "school near" material and actually match it (maybe slightly overperform, haven't checked) on "school far" material. PISA have come up with material that more resembles a traditional intelligence test and consequently the UK's performance closely matches its performance on said traditional intelligence tests (such as Raven's Matrices).

    This is useful for a number of reasons. Firstly it shows that our school system is pretty objectively terrible: it fails to teach what it should and, as noted here, particularly fails the bright kids. It might also encourage people to get to down to figuring out mathematically what percentage of our PISA deficit is due to having less intelligent children than East Asia, and how much is due to worse teaching and other non-genetic factors (and yes, of course it matters that our kids are less academically capable than would expected from their intelligence).

    All this guff about teaching "problem solving/21st century skills" is simply nonsense. You cannot teach these things: they are inherent to the individual. At most you can give them a chance to practice, but I doubt that has more than a tiny effect.

  3. On the off chance that they are being implemented then they can confront a ton of issues in their next life. Individuals ought not drive them to begin their life in the event that they would prefer not to. helpful link

  4. Summarizing is essential in Making your notes more concise. Try to outline in dot-points as if to instruct To somebody else paraphrase online generator is the best one paraphrasing service in my experience.

  5. You have post a good article. You have write so many thing that are really important for me. And you have added some pictures that make this post useful. I like your post reword my sentence keep writing.

  6. You might have submit an excellent posts. You might have compose numerous factor which can be important for me personally. And you also have got extra several images that produce this kind of submit beneficial. I prefer the submit read here retain creating.

  7. This is like the weird blog to read but looking also interesting. It is just my feeling but I know you have a good blogger with useful site and you know how to express your feelings by writing the interesting words. It is a totally fantastic way to express your feeling.