Wednesday, March 23, 2011

Redeploying Itanium Chip Designers -- Part One

The Itanium is Doomed

The Itanium is doomed; of that I have no doubt. That means that soon an entire ecosystem of architects and engineers will be looking for something to do. I suggest that we move them to a 'skunk-works' where they can recharge their batteries and reflect on lessons learned from their time working with the Itanium. [As an aside, I was informally asked if I wanted to work on the Itanium in the very early days of the project. I demurred, but mostly because it could only be as an employee and I worked for myself. But for the Grace of God ...]

Problems Itanium Had to Solve

The Itanium was intended to replace the x86 by addressing a variety of problems with the IA32 x86 architecture. It failed entirely in that mission.

In no particular order, here is an incomplete list of the x86 deficiencies as they occur to me:

It was 32 bits. Without getting into what 'bitness' might mean for a processor, it was not big enough.

Addressing more than 2^32 bytes was difficult or impossible (depending on circumstance). No 'flat address' space larger than 2^32 bytes wide could be addressed by one pointer.

It had a crufty old instruction set with a variety of 'hacks' to make it backward compatible over a number of generations spanning from the 8008 (released before the 4-bit 4004 and worked on by the same people). The nature of the instruction set made the chip awkward to program.

It did not have enough registers.

It had (at least for me) an odd addressing scheme and odd protection modes.

It was not well suited to multi-processor configurations.

It had bandwidth problems. Feeding the CPU with instructions from memory was about to become a bottleneck.

It made some compiler optimizations difficult or impossible.

It was not fast enough.

Problems Solved by the competition (x86-64)

In the years since the IA32 architecture was the last generation x86 architecture, some of the problems have been addressed in chips presenting the x86-64 architecture (such as Opterons, XEONs, etc), .

The chip is now 64 bits rather than 32 -- that is better.

It is now possible to address 'flat' address spaces much larger than 2^32 bytes.

There are more registers. That's also better.

I am not entirely sure where we stand with the addressing scheme, but for practical purposes, you can use a 64-bit flat space for most things that concern me. Messing around with segment registers is long behind me now.

With respect to the protection schemes, this is still messed up, in my opinion.

The chips are all now much better suited to multi-processor configurations. In addition, they all can support multiple cores. Both AMD and Intel will support 4 socket designs. AMD supports a 4-socket design with 12 cores per socket for a total of 48 cores. As of this writing, even a consumer grade machine can have 6 real cores or 4 that look like 8 cores with hyperthreading.

Bandwidth problems will always be with us, but the bandwidth situation has been very greatly improved through a variety of techniques.

Certainly EPIC style optimizations are still not possible. However, as in past CPU upgrades, modern x86-64 chips have a variety of built-in extended instructions that can be used to further optimize code.

Chips will never be fast enough, but the current generation is significantly faster than the chips were in 1989 when HP first contemplated building the EPIC based system that they began developing with Intel in 1994. In 1989, the problem they were solving (performance-wise) was an upper speed limit of 25MHz with a nominal limit of 1 instruction per clock cycle. Even by 1994 when they began in earnest, clock rates were hovering around 100MHz. Today, as this is being written, consumer systems can deliver performance in the 3.46GHz range across 6 cores -- effectively delivering more than 20,000MHz -- more 200 times the power of the chips the Itanium was supposed to best, (nominally) 25 times the power of the first Itanium released a decade ago. None of the top ten supercomputer sites use Itaniums. Most use Itanium's x86-64 rivals.

Business case is an EPIC Fail

In fairness to the Itanium, I think there may be lots of ways that a given off-the shelf Itanium box can best an off-the shelf x86-64 box. There *is* some merit in the EPIC notion and the chip accomplishes at least some of its goals for scaling well with certain types of workloads. However, for any workload I am likely to have, it is no contest and when making a business case for a purchase like this you are not comparing 'box to box', you are comparing 'cost per usage' to 'cost per usage'. On that basis, there is no ordinary scenario where the x86-64 does not flatten the Itanium. If you factor in the cost of re-writing your software on the Itanium before you can run it, it is hopeless. That last may seem extreme, but the Itanium has a very small ecosystem and the ecosystem has to somehow support the cost of that port.

The Good News

The Itanium, as I say, is doomed. That's the bad news. The good news is that those systems people are due to come free one way or another. That means that we have a team of experienced and battle-hardened veterans with (at least many of) the skills needed to solve the remaining problems with the x86 architecture. Just because they got it wrong with the Itanium does not mean that all of the ideas were bad or that their skills are lacking. Just because the x86-64 replacement of the x86-32 architecture does a better job of improving things than the Itanium does not mean it has actually done a good job or that no more problems remain.

In another posting, I will make some notes as to the x86 landscape as I see it and how I think it can be improved.

--- Copyright (C) Bob Trower. You may use this freely as you wish, but please attribute the work if you can ---

Is the Itanium Finally Dead?

The Intel Itanium was a still-born monster from day one. I had my doubts, as soon as I had any real information on that processor. When the first machines started coming out, it was clear that this was a total disaster.

The Itanium was quickly named the Itanic and the name stuck. If Google's numbers are to be believed, the use of the term 'Itanic' outnumbers the use of the term 'Itanium' about 20 to 1. The product has been synonymous with EPIC (pun intended) failure almost since day one.

The wonder is not that Intel is finally killing off the Itanium. The wonder is why it took so long and so much money to learn what should have been clear from the start.

I published an article about the ongoing CPU struggle between AMD and Intel in 2000 ("Why Should AMD Drop Mustang?" under the pseudonym 'DeepNorth'. In the article, I say:

"For the first half of 2002, Intel needs for McKinley to have such compelling performance advantages over 32-bit systems (and 64-bit Hammer) that it is worth doing a software rewrite for the majority of developers. My 20 years in the industry tell me the chances of that happening are virtually nil." [McKinley was the coming next generation Itanium]

I was not expecting much from McKinley anyway, but it underwhelmed even me when it was released. Nobody I worked with was writing stuff for the Itanium. I do not remember even seeing an Itanium box in the wild. They exist, but in tiny quantities compared to the x86.

In the closing footnote of that article, I say the following:

"I don't believe that there is much argument that the 'Itanic' is a failure, even from Intel. Will this processor ever be anything other than a historical curiosity?"

That was over ten years ago. About two years ago, it was still 'steady as she goes' from Intel and HP, even though the processor had been all but dead for most of its existence. This ship would have sunk long ago had Intel, HP, Microsoft and similar big players not been madly bailing water all these years. However, over the past year, the Itanium has begun to take on too much water even for industry titans. It is sinking fast now. Even Intel has dropped C++ compiler support for the Itanic. This week, Oracle delivered the coup-de-grace by announcing that it was discontinuing support for the Itanium. HP is the only major player that has not dropped the hot potato that is the Itanium. They can't hold it forever.

Possibly the only thing that Itanium had going for it was the fact that it could run Windows. It can still run Windows, but not for long. Mainstream support for the current Itanium version of Windows ends in 2013 and 'extended support' ends in 2018.

Dan Reger, Senior Technical Product Manager for Windows Server for Microsoft has this to say about the Itanium vs the x86 chips:

"Microsoft will continue to focus on the x64 architecture, and it’s [(sic)] new business-critical role, while we continue to support Itanium customers for the next 8 years as this transition is completed."

Hardware needs software to breath. The Itanium's oxygen has been cut off.

I predicted that the Itanium was doomed a decade ago. That was in the face of a firm commitment of billions of dollars by major industry players to make Itanium work. Nothing at all has been added to the 'plus' column since then and many have been removed. Any remaining entries in the 'plus' column are either trivial or irrelevant.

The Itanium is *still* not actually dead, but it has been taken off life support. It is just a matter of time. I think it is reasonable to say that it will be official soon, probably before the end of Microsoft's mainstream support in 2013.

Having programmed in assembly on both x86 processors and non-x86 processors, I have to say that I am not fond of the x86 architecture. Nobody would be happier to see it replaced than me. I think it will be replaced. In fact, it has arguably already been replaced with a less horrible (but still horrible) version of itself.

The Itanium was the wrong answer to the wrong question. It never would have been a success. That is a shame because it was attempting to address very real problems with the x86 architecture and many of the worst of these problems remain.

For displaced chip designers who might find themselves on another team, I will try to put up a wish-list from a long time programmer (me) that should be food for thought.

--- Copyright (C) Bob Trower. You may use this freely as you wish, but please attribute the work if you can ---

Friday, March 11, 2011

What's the deal with the updates

Every time I attempt to load anything it has to spend a whole bunch of time doing updates. It is nice that software is so diligent in keeping up to date. I just wish it would find a way to do it without bugging me.

Here's one of the most annoying. I've seen this a few times when loading FireFox:

It is strongly recommended that you apply this update for Firefox as soon as possible.

This has been coming up for a while now I always click next, and am presented with a message:

The license file for this version could not be found. Please visit the Firefox homepage for more information.

Sigh.

This can be fixed, but it is *way* annoying and the fix is not something an ordinary PC user would be comfortable with doing.

The short answer is to reboot and cross your fingers.

There is a very long procedure that is more bothersome than just downloading the whole thing and installing a new copy.

If the tortured procedure above (I am not going to put you through it) does not fix it, the last recourse is to download and re-install a new copy.

The Mozilla team is generally pretty good, but they occasionally crash and burn in a big way. This is one of those times.

Thursday, March 10, 2011

Windows 7 Backup does not work

Windows 7 backup often fails for a variety of reasons. You can usually troubleshoot these by the error messages that it gives you. It is torture, but it can be done. However, doing both a successful backup and then a subsequent restore if your system fails is so unlikely as to be effectively impossible.

Here is the simple rule for Windows backups: they only work when you don't need them.

If you look on the web for some assistance for backup and restore you will find precious little information. What you do find is all but unusable by anyone who is not an expert.

If everything is well and all goes smoothly, you can use instructions at sites like dummies.com. Here is a page on a vanilla restore from an existing backup:


I am making this post because I simply *had* to use Microsoft backup on Windows 7 and I *had* to restore that backup. It did not work.

There are so many things that can fail with Windows backups that it is impossible to itemize them here. I will just offer up the solution that allowed me to restore my files.

The files had been backed up to an external USB drive. This seems, from what I can tell, to be the only really viable method of backup for most people. It should be dead simple. Most common scenarios would have been tested if anyone seriously expected backups to both be made and then restored. Since Microsoft has probably something like 100,000 or more of its own desktops and tens, hundreds or thousands of millions of client desktops, you would think that a simple scenario would have been tested.

Here is my scenario: backup to a 2 TB USB hard drive and then restore from that drive shared on the network.

The backup was hampered by the usual errors. One that comes to mind gives the error "the operation timed-out before the shadow copy was created". It helpfully suggests to try another time. After trying off and on over a period of days I was finally able to get it to run successfully. There is nothing wrong with the drive or anything. I routinely move about gigabytes of data on that drive and this is the only operation that dies on that disk. Another error encountered was the system being unable to find the backup location when attempting to do an incremental backup to the last set. The fix for this is to re-run the backup utility and choose the drive and get it to try finding the backup a couple of times. Sometimes you have to also change the permissions of the backup directory and files. This is well beyond anything that it is reasonable to expect from an ordinary PC user.

When it came time to restore my hard won backup (it took literally days), it came up with an error:

Windows was not able to find any backup sets.

This is because it only expects to be restoring to the same machine from a prior backup and if you are restoring to a new machine you don't have any. It does not bother looking and it does not bother telling you that you should look either. So ... unless you have the backup on a network share so you can browse for it, you have to start looking for the files manually and inspect them manually to see what (if anything) can be done to repair them.

If the files are on a network share, you are not out of the woods. What I got when I went to the share with the backup on it was the message:

"Windows was not able to find any backup sets"

and the chipper "Please select a different location"

There is no meaningful help offered anywhere that I can find to solve this issue. I definitely had a backup that the originating machine could see. I even went back to it to do a test restore of a few files to make sure it was real.

The only thing that I could think of was that it was not visible somehow due to permission problems. I went back to the originating machine and carefully took ownership of the files as 'Everyone' with full permission and then removed all the other permissions. This was after a few false starts with permissions that were more sensible. Exasperated, I decided to give up and just give the world all permissions to make sure that was not the problem. I know from other forays with this stuff that permissions *can* be the problem. In this case, though, it was not. That is, it *was* permissions, just not on the backup itself. It was somewhere else.

The last thing that finally got the backup and its files visible was to change the permissions and ownership of a file in the root of the disk used for the backup:

MediaID.bin

As with the files themselves, I changed this to world readable, world writable (permissions assigned to the user/group 'Everyone').

That last bit finally got the backup visible and I was, after days of misery, able to restore the backup.

This helped me, but it is not a fix. I am at a loss as to why nobody will 'fess up' that you cannot effectively backup and restore Windows systems. It is theoretically possible, if the stars align themselves, but for any practical purpose it is not possible. It has obviously not been very extensively tested by Microsoft and is not in widespread use. This is one of the shameful secrets of the computer industry. Everywhere you look, you are glibly told to do a backup before trying anything to correct problems. Nobody mentions how exceedingly difficult it can be.

A backup that you cannot restore is useless as a backup.

When you are in the position of attempting to restore a backup you usually do not have access to the original system. You are attempting to recover from a failure of that system.

If you are in the most dire position of attempting to recover from a catastrophic failure, chances are good that other things have gone wrong. This is not the time for an elaborate fussy, fragile system that breaks the instant something is not exactly as anticipated by the designer. Certainly it is not the time for an elaborate system that has not even been fully tested. This is the time for a dead simple system that allows you to find and open a single file that has whatever it needs to restore your files. Requiring that the system already know about the backup and it have a special file in the root of a hard disk and must be stored in a certain place, etc is just downright crazy. It is a sure-fire recipe for failure.

I have been working with systems for decades. It is exceedingly rare for me to lose anything and I never lose anything important. How is this accomplished? I make multiple copies of files. The number of copies is proportional to their importance. Mission critical client data is generally backed up across a number of geographically distributed systems.

The disaster recovery cycle is torturous in the extreme. I expect that most people end up losing most of their stuff from time to time because there is no reliable way for them to prevent it.

I am surrounded by enough systems that they *do* fail and we are required to bring them back up. Typically this involves a fairly time consuming cycle of re-building or replacing equipment, re-installing software and then restoring files and settings from copies.

Windows backup *is* used successfully, but it has to be very carefully and conservatively set up, has to be dead vanilla and has to be periodically tested (you have to do a test recovery). Even then, any critical data needs to be backed up separately and the backup has to account for a wide variety of settings that depend upon the version of Windows being used, the particular software being used and the configuration of the source system (the thing you backed up) and the target system (the thing to which you are restoring).

If you are very careful, you will note one of the signs that the backup/restore facilities have not been extensively tested (by anyone anywhere). When you are choosing the network location to *restore* from, the dialog has the following embarrassing caption:

"Select the folder where you want to save the backup"

Imagine if, every time that you went to load a file, the system asked you to choose the file to save. Each time you saw that, you would have to convince yourself that selecting the file would not end up overwriting your only copy. It would be irritating in the extreme. You don't see that in major products because they are being used all the time. People would quickly catch the problem. The fact that the above caption is messed up like that means that even though it should be something that has been used hundreds of millions of times it likely has only been used a few thousand or less.

The bottom line is that you should test however you are going to backup and subsequently recover. It is possible to backup and restore using Windows' built in facilities, but it is not very likely. You should not depend upon it for anything important.

Microsoft should do penance for all the misery it has caused. Here is a way that I think they could make amends and cure this problem once and for all: Fund an open-source project to develop a proper backup facility that would be able to backup and restore files from/to the majority of commodity systems. It should be able to save, for instance, a backup from an old Windows 95 box and restore that to a modern Linux box. This is not simple and for some things, there is no apples-to-apples equivalent of source/target. Linux, for instance, does not have a registry.

I do not expect that a complex installation of a Microsoft IIS web server and development environment can be backed up and restored to a Linux system. However, I *do* expect that a Windows 7 system can be backed up and restored to another Windows 7 system.

I am aware of challenges such as the registry, DLL hell, disk geometry, drivers, etc. It is a story for another day, but all that stuff is horribly broken. The problems with these systems goes back a long way. A backup program cannot fix everything, but it should be able to fix many of the things that currently make it so difficult to maintain systems.


Monday, March 7, 2011

Our education system is broken

I have struggled over the years with a persistent feeling that there is something profoundly wrong with our system of education. I expect that everyone wonders, from time to time, if we might do things better. I am increasingly of the opinion that we could do things a lot better. However, I expect that this will require some very large changes to the way we do things now.

I am reluctant to criticize the way people working in a field conduct their business. I have spent a significant portion of my career examining how people do their work and I have invariably found that people are skilled, knowledgeable and conscientious. They know what they are doing better than an outsider would. Often, even though it seems from the outside that things do not work as well as they should, a closer look will reveal that the problem domain is larger and more complex than it looks and the solutions being pursued by working experts in the field are mature and effective.

I have spent a lot of time in our educational system as a student, a teacher, a parent and an advisor. I have seen the system 'up close' and I continue to feel that something is just not right. It has been my experience that when things appear to be a bit 'off' in an industry, they are not. An apprehension that things are being poorly executed is usually wrong. However, in this case, my gut will not let me alone. I cannot shake the feeling that our system of education is simply broken.

I will perhaps write about this at greater length some other time, but for now, I am making this entry here to take note of something that occurred to me after I encountered yet one more thing that tugged at my 'gut' instinct that something ails our educational system.

Below are two epigrams leading into the doctoral thesis titled 'Architectural Styles and the Design of Network-based Software Architectures' by Roy Thomas Fielding.
What is life?
It is the flash of a firefly in the night.
It is the breath of a buffalo in the wintertime.
It is the little shadow which runs across the grass
and loses itself in the sunset.
— Crowfoot's last words (1890), Blackfoot warrior and orator.

Almost everybody feels at peace with nature: listening to the ocean waves against the shore, by a still lake, in a field of grass, on a windblown heath. One day, when we have learned the timeless way again, we shall feel the same about our towns, and we shall feel as much at peace in them, as we do today walking by the ocean, or stretched out in the long grass of a meadow.
— Christopher Alexander, The Timeless Way of Building (1979
Life is an ineffably nuanced experience and it is profound. When good, it just feels 'right'. It does not feel quite right in our towns and cities, but it could, and someday perhaps will.

Whatever else we might say about our kindergartens, grade schools, high schools, colleges and universities, they do not feel quite right to many people. They are not natural and comfortable. For the most part, the things we like about them exist in spite of the schools, not because of them. For most (admittedly not all), going to school is a means to an end, not an end in itself.

Were we to get it right, formal education would likely 'feel' much different than it does now. As it stands, the school system is a foreign thing. For more than a hundred years, we have lived with this contrivance and it has never felt right. I do not think that it can because it is predicated upon a variety of unnatural premises. Everything about the system is difficult, awkward and forced.

Those epigrams eloquently convey the idea that if things are as they should be, they 'feel' right. This implies that if things do not 'feel' right, then they are not right. They are not as they should be. That added a small bit to my conviction that things are wrong. Whether or not I can articulate the precise causes of our educational system going awry, I can tell that it is not right because it simply does not feel right.

For whatever reason, this finally caused something to settle firmly into place for me. I can finally reconcile my instinct that educators cannot be incompetent with the fact that the educational enterprise qua educational enterprise is a failure. You would think that if there were a better way there would be at least one jurisdiction to discover it and break from the pack. Since they appear to do it more or less the same way at least across all of North America, it would seem that this must be as good as it gets. They cannot, reasonably, have spent a century perfecting a system of education and be complete failures at it. Here is the flaw in that line of reasoning: It assumes that the genuine mission of our 'industrialized' school system is delivering useful education. We are assuming that the true goal of the people administering our educational systems is the effective delivery of a quality education to their students. If that were the case, I have no doubt that we would have an extremely effective system of education and we would know it.

If education is not the primary goal of the educational establishment, then what is it? To answer that, as with many other things, it is best to 'follow the money'. Where does the bulk of the money put into the educational system go? It goes to pay people within the system. That makes sense. It has historically been a labor-intensive pursuit. However, even if we assume that people are doing their best to ensure those salaries are the best way to deliver education; you can see that the 'tidal force' of all that money is likely to have some effect.

The people and other entities that have been involved in the school system do not effectively have education as a primary goal. Their primary goal is to serve the interests of the people running the system. The people running the system are not the students, their parents or any other similar primary stakeholder. The people running the system, I can see from close-up in my community, are the administrators of the system. The powerful teachers' unions prevent them from putting teachers at a disadvantage, but everyone else is fair game, including students, parents and their representatives on school boards.

I am not saying that educators and administrators show up every day concentrating mostly on furthering their own interests. As individuals, I am certain that the majority of people in the educational enterprise are focused primarily on the nominal mission of delivering education as well as they are able. As a former teacher in the college system, I know that my personal focus was exclusively on how best to deliver education. I cannot speak for others, but aspects of the teaching enterprise were frequently the topic of talk around the water cooler.

I also believe that other entities within the educational enterprise, as much as one can impute intent to them, are similarly engaged with the task of delivering education as well as they can.

The problems with our educational enterprise are structural and systemic. When it comes down to crucial funding decisions, when the rubber hits the road, powerful forces are at work to maintain the status quo. To the extent that changes take place, they have a tendency to act in favor of improving the lot of administrators and actors such as teachers. Once upon a time, teaching was near synonymous with poverty. Now, teachers are amongst the most financially powerful players in our society. This may have aided in getting better teachers and ultimately delivering a better education, but it would be foolish to think that the main goal of negotiations between teachers and school boards was the welfare of students.

One of the things that worry me is this: We devote time to helping the school with things like fund-raisers and still spend significant amounts of time teaching academic subjects to our own children. The school has been steadily abdicating responsibility for its primary mission since I was a student. I spend more time doing homework with a single one of my children in a year than I did in my entire school career from grade school through University. They are, I think, better than I was at that age, but at what cost?

One should not come to management with a problem without proposing a solution. I cannot reasonably propose an entire solution to such a huge problem. However, here are some ideas:

We should admit that our system is sub-optimal. The first step to solving a problem is acknowledging you have one.

We should realize that the problems are systemic and require radical changes. Band-aid approaches will not work. Slower, more evolutionary changes will likely not work either. We need to re-think the entirety of the mission here and to restructure society so we more effectively support that mission. Right now, it almost seems as if the nominal mission is to produce a small number of joyless pedants at any cost and the hidden agenda is to enrich the segment of society that has gained control of our educational system.

We should alter the system completely, so that it properly serves the realistic needs and goals of the community.

Here are some ideas that I feel could help in developing a solution:

Even the stated goals of our system are wrong. We focus too much on straight academics as if the goal of the system is to turn everyone into professors. We attempt to place the school at the center of the community. That is fine, but not if we remove control from students and parents. That defeats the notion of community. Either the community has to be much more involved in the schools or the schools have to integrate themselves back into the community. More people should teach and more teachers should participate in other productive activities.

Beyond over-emphasizing 'hard' academic subjects like math and science, we under-emphasize music, dance, painting, sculpture, language and more. Vast areas of human endeavor are marginalized needlessly. This causes things like dance to become almost a guilty secret and interest in going to higher levels in activities like sports is left to chance and the resources of the student. If it is a human activity and it requires time and effort to master the required skills then it should take a place of honor amongst the teaching activities of the school as much as any other. What is an ordinary person more likely to use in actual practice: Dance or Calculus? It is not even a contest. If anyone should feel a little sheepish about learning a subject just for the heck of it, it is the mathematician. The dancer is more likely to give back in practice. There are plenty of dance studios around and plenty of places to dance. There is, to my knowledge, no such thing as a 'math studio' and there are not many people lining up to see a Broadway show with a name something like 'Integrate -- experiments in modern interpretive math'.

It may be that some core literacy subjects are required to learn other things. If that is the case, we should accumulate them while pursuing the other things. That will force us to be economical in delivering genuine skill as needed.

It may be that some subjects are just no fun but we need people to do them in order to support our industrial economy. That is fine, but if that is the case, we should say so and make financial rewards high enough to induce enough people to learn them without further coercion. Beyond that, since it requires some level of coercion, we should recognize it for what it is -- a necessary evil -- and devote appropriate resources to eliminating the necessity for that evil.

Note that coercion is real enough when it comes to education. In at least most of North America, students and their parents are required by law to ensure that the students attend school. Whatever we are doing in our schools, it is sufficiently noxious to some that they would not use them if they were not legally obliged to do so.

A subject that is difficult and unpleasant needs to be justified and the more difficult and unpleasant it is the greater should be the justification. I would say that many subjects in school such as History, Geography, Algebra, Geometry, Calculus, Physics, Biology and Chemistry are over-taught to people who neither need nor desire that level of knowledge in that subject. The level they achieve is not even retained. For the most part, it is just a lot of time and misery for everyone to no purpose. If nothing else, it should offend you because it is wasteful and sloppy.

One size does not fit all. We have a structured system of grades and subject matter that ensures mediocrity. It does this by stifling those with the greatest combination of ability and enthusiasm and overburdening those with the least.

We should decouple subject areas and areas of competency from one another. People should be able to proceed at the pace that is best for them. An ability at the 'grade 6' level in baseball is unlikely to be entirely matched by an exact ability in mathematics at the 'grade 6' level. By forcing everyone to take all subjects in lock step, you almost guarantee that every single person fails to meet their potential. They will be left with unrealized progress in areas where they are stronger while still feeling overburdened because they are being forced along at too great a pace in areas where they are weaker. Our choice should not be between the three mutually exclusive options of being promoted an entire grade with one's peers, being promoted an entire grade ahead of one's peers or being held back. None of those options is either good or necessary. People should proceed as quickly as they have ability and motivation in any competency. They should take whatever time they need to master a subject and should only master subjects they either need or want.

Our school system currently sorts students into groups representing failure, under-achievement, 'average' and over-achievement. Only one of these is seen as being something in which we can take pride (over-achievement) and arguably even that is not an admirable goal since most of the 'over-achievers' suffered at least a little to reach that level.

Instead of the current sorting as above, we should organize activities around competencies and should have only three categories: unlearned, learning, mastered. We should not be making some students 'better at grade 6' than others. Grade 6 is not even a subject area or competency. It is meaningless in the wild and we should not be trying to give it meaning.

We should abolish grades. To the extent that we track development in terms of age, this should be switched to doing it by month rather than year. Our current system places people who are sometimes effectively a year apart in age in direct competition with each other. If nothing else, this is unfair. It creates, unnecessarily, a 'lottery' of winners and losers based on date of birth. Life is inherently unfair, but that does not mean that we should not attempt to make a level playing field when we can.

Education may always be somewhat labor intensive. It is, if nothing else, something of a social enterprise, and that requires people to interact as students and teachers. However, not all of that interaction has to be between paid 'teacher only' teachers with college credentials and unpaid 'student only' students.

We should train our students as teachers from the beginning. As soon as they are able to perform useful work as teachers, we should set them to work a part of their day as teachers and pay them accordingly. That would ease the ultimate financial burden on families by lowering the total salary costs required by schools (a grade 6 student is not likely to be as expensive a hire as a college graduate) and providing some modest additional income.

Studies have shown that 'peer tutoring' of, say, grade two students by grade four students will have a net beneficial effect on the grade four 'teachers'. That is, if we take two of a student's six hours a day at school and set them to work as teachers, they will 'net' learn more in that six-hour day than they would have if they had spent the entire day exclusively in the role of student. The grade two students do not perform any worse with peer tutors than they do with credentialed adult teachers. Why bother spending the extra money?

We have, I think, a tendency to teach many things that are of little interest and little use. This is extremely wasteful. It discourages students. It decreases their 'buy in' to the educational enterprise. It takes time away from productive activities. Ultimately, they don't learn the subject anyway. It just means that instead of knowing one subject well, a student ends up knowing that one subject poorly whilst enduring and forgetting another subject.

It would be nice if all of us had the ability, training and interest to appreciate mathematics. However, we just do not. We should not be putting half the class through torture to learn, temporarily, how to do algebra. We should be integrating normal number sense into useful activities (that both student and teacher find useful) for most people. For some, mathematics is fascinating and worthwhile in itself. Instead of wasting classroom time forcing disinterested students to learn things they neither need nor want, we should devote additional resources to those who are keen to learn more.

We should engineer success for all the participants in the educational enterprise. Noxious stimuli do not result in maximal behavioral change. The best way to alter the behavior in animals is to place them on a variable intermittent system of reward based upon their behavior. Individuals will do best if we reward them for effort. It is up to us to engineer the situation where individuals have to 'stretch' to reach their goals, but have enough success along the way that the process itself is rewarding.

Put another way, we should do our best to determine what a student is capable of doing, encourage them to seek that goal and then reward them for reaching it.

All goals should be ultimately reachable, but we should focus our attention on the journey, not the destination. Rather than striving to spend only moments on the podium getting our diplomas every few years, we should be striving daily for all manner of small progress toward competency. We should be able to point daily to an enjoyable effort and weekly or monthly toward meaningful progress.

I honestly do not know if a system that only rewards the top three places in a competition with a hundred people is optimal. However, if I had to place a bet right now, I would bet against it. We currently have a system where top athletes overreach themselves, to their ultimate long-term detriment, and bottom athletes give up and don't participate at all. Everybody but the people at the top is 'a loser' and everyone at the top injures themselves to get there. Nobody wins this game.

The top few individuals in a given discipline appear to contribute disproportionately. Perhaps it is true. It may be that we are best served by creating extraordinary rewards for these individuals so that they will be as productive as possible. I am suspicious of this, but it is definitely worth testing. If it turns out to be true, I doubt many would mind giving a few of the top people more than an equal share. A rational way of reflecting this disparity in abilities would almost certainly be better than the current state of affairs. We currently have a small class of superstars with control of a wastefully disproportionate share of resources and a large underclass that wakes up facing insuperable challenges every morning and going to bed as failures every night.

As a society, we tend toward rewarding the fortunate and punishing the unfortunate. This is probably not the optimal way to proceed. It certainly can't be the most humane.

The process of education should be as natural and organic as learning to walk or (better) ride a bike. Almost everyone can learn to ride a bike, but for some it is effortless and for others it can be difficult. The age at which people learn to ride a bike varies much more than the age at which we try to teach them mixed fractions. We think little of a disparity of years for riding a bike, but currently we have a very hard deadline for teaching mixed fractions. If you don't get mixed fractions when we try to serve them to you, we don't really give you a 'do-over'.

The current reaction to complaints about our school system has been a shift toward 'high stakes' testing. This, in my opinion, is worse than useless. Half of what we are testing is not that important anyway. For things that *are* important, we owe the students more than putting a gun to their head and insisting they choose between misery and failure. We have responded to complaints of poor outcomes from our educational system by making sure that our schools are capable of reporting success on high stakes exams. What do we get from that? It is no secret that teachers begin teaching to, and students begin studying for, the test itself. Do we even get successful test-takers? Do we just get systemic cheating in the preparation for, administration of and reporting of exams?

In recent times, we have come to see primary and secondary schools only as means to an end. The entire enterprise is aimed at getting people ready for their 'real education' either at post-secondary institutions or on the job. Youngsters are trained through secondary school with the aim of 'getting in' to either good post-secondary educational institutions or jobs. There is an insidious assumption there that may have once had a basis in fact but is now yet one more evil in the educational system:

We have an artificial scarcity of places in 'top-quality' institutions and jobs. Enrollment is strictly limited and you have to be somehow lucky to get into them. In addition, institutions that are more exclusive are prohibitively expensive for a person of average financial means. As of this point in time (2011), there is no need for the quality of education at either an 'Ivy League' school like 'Harvard' or 'Yale' or a State University to differ in any significant way. In fact, there may not be that much need for these institutions at all, at least for most undergraduate studies. We should be attempting to limit our 'credentialing' institutions to validating and issuing credentials and we should insist that they do this with the type of economy that our currently over-educated highly automated society allows. No (at least purely academic) degree should cost more than can comfortably be borne by the public purse. That is, the limiting factor should be interest and nothing else. We can do this, of that there is no doubt.

[As an aside, I think the road to making 'factory degrees' (my coinage(?)) respectable is to make them slightly better than ordinary degrees and more uniform in terms of the quality they imply. This is easily accomplished. All you have to do is make it highly cost effective, validate to a higher standard and discourage/disallow equivalence. That is, you can't say you have the equivalent of that degree. Either you have it or you don't. Eventually, when a four-year 'factory degree' is universally known to signify the highest standard of quality what will happen is people from ordinary colleges will opt to test for and get the 'standard' degree. Over time, people will skip the old style degree and go straight to the standard. How students are educated is irrelevant. It may be that the storied grounds of Harvard are more efficient at delivering an education somehow. It doesn't matter. The only thing that counts are the results and the only thing the credentialing shows is results. If Harvard is that good it can sell its secret formula for whatever the market will bear. I predict that some time in the next twenty years the floodgates will open as schools like MIT come to offer higher quality, lower cost degrees via distance learning. If someone exceptional can travel to the end of a Doctorate for $10K in five years vs. spending $250K over more than a decade, the smart ones will opt for publishing for a half a decade longer and driving better cars rather than starving for the whole decade. At some point, the balance tips so that the more expensive degree is not only a poor asset but may become a positive liability.]

In addition to an artificial scarcity of educational opportunities, we also have what in my opinion is an artificial scarcity of quality jobs. The reason I say it is artificial is this: we currently have a significant proportion of all of society unemployed -- likely at least half of all adults are underemployed. Despite that unemployment, they are (mostly) fed, clothed, housed, educated and entertained. In essence, they participate as second-class citizens and are reminded daily that their ability to carry on is only at the sufferance of their betters. How long has this been the case? For likely a hundred years or more. Things have gotten worse in recent years, but at least during the great depression of the 1930s, we had massive unemployment. We have structured society so that some members are forced to occupy most of their time in paid pursuits away from their families and some are forced to remain idle and live in poverty. This is not some recent mistake. It is planned into our economy and has been that way for at least a century. It's how we roll.

Not the exact numbers, but the idea is there: half of us are overworked and have beautiful homes we never see and half of us are idle and live in less beautiful homes from which poverty allows no escape. Rather than force some to spend 60 hours of their time working to make money they cannot enjoy and others to idle their days away in poverty we should shift some of the work and wealth from one group to the other. Change the notion of 'full time' to be 30 hours and shift the other thirty hours from the overworked group to the group that needs work.

We have a shortage of labor-intensive medical care and a shortage of work. How can that be possible any longer than it takes to train a cohort of new doctors? Chances are good that we can't take all the new doctors directly from the unemployed, but I have no doubt that we can definitely take enough to satisfy our demand for doctors from the underemployed. I do not think that we have any shortage of people willing to become doctors. I think we have an artificial scarcity of medical training.

More and more time is spent by individuals doing 'self-serve', while we have people who want to work but can't. There is a disjoint here. Somehow, we have distorted things so that busy people are annoyed with all sorts of self-serve tasks and poor people are left involuntarily idle. We do not have to make one group directly serve the other. Perhaps we could all invest in automating things. We could train people to participate in the next generation of automation, for instance.

I don't think that we can stop elements of society from fractionating out one way or another. It was ever thus. People will create and institutionalize 'cliques' of one kind or another. Of necessity, some of these have to allow a disparate membership joined essentially only by their membership in the clique. Also of necessity, there will be rivalries and some layering will occur such that some clusters are perceived to be more valued than are others. However, I think most people would be fine with this and the more noxious characteristics can be mitigated by creating a general cultural ethic that helps everyone to contribute as well as they can and values every contribution for its relative effort from the contributor as well as the absolute value to society. That ethic, in my opinion, will create the maximum happiness and prosperity for everyone.

Our school system is very badly broken. The fix is a radical overhaul, not just of the system, but societal expectations and the whole way we structure our communities. It is not that we cannot improve upon education. I am convinced we can. At the very least, we can get the same results at a lower cost and in a more humane way. The problem is that the current system is very much entrenched. The whole of modern society is at least partially structured around the existing system. To fix our ailing educational system, will require sweeping and difficult changes, but I think we will see the benefits rapidly and I think it will be well worth the effort.

Javascript webp to png converter

[Done with programmer's assistants: Gemini, DALL-E] OpenAI's DALL-E produces images, but as webp files which can be awkward to work ...