HIGHER EDUCATION BUBBLE UPDATE: Khan Academy: The Videos Are The Public Face, But The Real Action Is On The Backend:

Using math and computer science concepts decidedly more advanced than most of those in Khan’s video library, the Khan engineers have trained the website’s exercise platform how to predict, with startling accuracy, how likely it is that a student will correctly answer the next practice problem — and whether that student will be able to solve the same type of problem a week, two weeks, and a month later.

They do this by accounting for hundreds of data points that describe, in numbers, the entire history of the relationship between a learner and a concept.

“If [a user is] logged in, then we have the entire history of every problem they’ve done, and how long it took them, and how they did,” says Ben Kamens, the lead developer at Khan Academy. “So whenever anybody does a problem, we see whether they got it right or wrong, how many tries it took them, what their guess was, what the problem was, how many hints they used, and how long they took between each hint.”

The Khan engineers are also working to tweak the exercise platform so it does not confuse genuine mastery with “pattern matching” — a method of problem-solving wherein a student mechanically rehashes the steps necessary to solve that type of problem without necessarily grasping, conceptually, what those steps represent.

Pattern-matching is one of the human brain’s most basic learning tools, Kamens says. It is the sort of useful imitation that allows toddlers to learn how to use language without first learning how grammar works. But there is a difference between imitating problem-solving procedures and mastering the logic undergirding those procedures, Kamens says. Getting to that level of understanding, he says, is probably what determines whether students will remember how to solve a problem after the test is over, after a course is over, and — most importantly, in Khan’s view — once their formal schooling is over.

Khan has half-joked that his ideal assessment model is having professors ambush their students in the hallways with random questions, months after the student has passed the exam, and revise their score based on whether they’ve kept their chops. At Khan Academy, that half-joke is half-real. At a time when students are always within arm’s reach of a computer and a wireless signal, “mechanic practice schedulers” can spring questions on students at intervals to gauge how well they remember how to do certain types of problems.

One of the most interesting parts of Andy Rosen’s new book, Change.edu, is the discussion of how for-profit online schools are using similar techniques. This may turn out to be a killer app as against traditional education.

Plus this:

College students do not graduate with a firm enough grasp of the skills — particularly in STEM (science, technology, engineering, math) fields — that they really need to land good jobs, he said. As a result, the credential colleges use to signal the competence of their graduates, the college degree, says very little about what its holder actually knows.

In a conversation with Inside Higher Ed last week, Khan expressed some ideas on how to improve the signaling quality of academic credentials. Under the current regime, a degree from a college amounts to something similar to an acceptance letter from that college, he says. And that is not ideal for employers.

“We find a lot of college grads with high GPAs that have been exposed to many things … but even in their purported majors, they have a pretty weak grasp of” essential concepts, Khan said. “It’s almost like you view them as a blank slate, and the most impressive thing about them is that they got in to University X.”

In other words, the current price of a college degree is not just the balance of four years’ tuition; one must also consider the cost, to students and employers, of the ambiguity hanging over what the degree actually means. One root of the problem is the fact that the college degree is issued by the same institution that is in charge of setting, and enforcing, the standards of that credential, says Khan, who holds four degrees himself. This is tantamount to investment banks rating their own securities, he says. Meanwhile, the accrediting agencies that are in charge of making sure those “ratings” are legitimate do not currently focus on what students coming out of those institutions measurably know.

If you read the whole thing — and you should — it sounds like Khan is moving toward the sort of third-party credentialing solution that I have advocated myself in the past. That’s a real killer app.