We use college for too many purposes

In my view, our focus on college stems from our habit of using college for three related but different purposes:
– general intellectual training
– elite selection and preparation
– job-specific training

If we were to separate these functions, we would probably be better off.

To begin with, I doubt there would be much of a demand for post-secondary schools focused on the first function (general intellectual training,) but perhaps some of the hardest-core intellectuals would be into it. Becoming smarter and more knowledgeable in general is a worthy thing, but it should be possible to do it on the cheap. I doubt we’d subsidize it much, if at all.

The second function (elite selection and training) is why I think we focus so intently on colleges as they are now. We use them as gatekeepers to high position, the big jobs with high status and usually high pay. But does it really have to take four years, or even more, to sort out the capable from the nearly capable? The military doesn’t run Officer Candidate School for four years; it’s more like half a year. I can accept that selecting the people who are going to (or are likely to) run the show is worth doing carefully, and that there are some things worth teaching them. And of course since people are status-hungry, competition for position at whatever Social OCSes we run is going to be keen. But whatever else these institutions are, they are going to be small, because the elite pretty much by definition has to be small. Separated out, I would expect this function could be served in a year or so, and might enroll 10% of each cohort. Maybe 20% if we’re feeling exceptionally egalitarian and want to cast the net wide.

And finally we arrive at the job-specific training. This is what most people actually need, but it’s also what we’re not doing a particularly good job of delivering. To begin with, there is no particular reason this function has to be combined with the other two. We don’t need to educate the folks who want to be writers and editors together with the folks who want to be data scientists and engineers. Their job functions have very little overlap, meaning their curricula should have very little overlap, so their education could easily be done in separate institutions. And job training programs can be diverse. Some jobs are simple and require no training at all. Others are very complicated, and can easily take close to a decade of training to master. And if we were to separate the three functions, only a few of the job training institutes would look and operate much like traditional colleges.

But ultimately, we are so focused on college, and so willing to spend money on them, because we are confounding three quite different purposes that current colleges serve: general intellectual training, elite selection and preparation, and job-specific training. If we are to separate the functions, the first one would probably mostly wither, the second would be short and specific (and hence not too expensive), and the third would have its proper place as the destination for nearly everybody.

What Star Wars VII should have been

I thought The Force Awakens had some strong parts, like the attack on Death Star III and the engaging trio Rey/Poe/Finn. But it wasn’t the story I wanted to be told. I didn’t need to see A New Hope again. I would much rather have seen the original characters age and grow, and face new challenges.

Don’t show me Leia leading a rag-tag rebellion. Show me an ageing professional politician wrestling with the impossible demands of a thousand worlds.

Don’t show me Han working as a smuggler. Show me an old warrior bored out of his skull serving as Minister of Whatever, yearning for something else, anything else.

And throw in a Luke who after decades of devotion to the Force sees everything from every perspective all the time, to the point that he is barely even human any more.

Then have the New Republic face some hammer-blow of a challenge, where things go so badly wrong that these senior figures are inadvertently on the front lines, letting them have one last big adventure before the fight is taken up by younger hands.

That would have been a film worth watching.

The Secret

The girl was on time for her appointment. She walked into my office and sat down in my guest chair.  Girl? Woman? She was younger than my daughters, anyway.

"Thanks you for agreeing to see me."

"We don't hear from your organization often. Your request was, shall we say, intriguing."

The slightest of smiles. "We have a problem for you."

"We specialize in solving them. How can we be of assistance?"

"We have a secret that must be kept for a long time, and then revealed to everyone."

"What sort of secret?"

"I am not allowed to disclose that. Not to you. Even I know only a portion of it."

"I'm going to need at least a hint or two."

"Very well. To speak very metaphorically, we have struck a spark. The spark will grow into a fire. The fire will burn a land bare. And in the bare land, a precious thing will grow."

"And of these the secret is ... ?"

"That the spark has been struck, and the fire is coming."

"And why does that need to be a secret?"

She looked away. "Because striking the spark was nearly unconscionable, and if people know the fire is coming they will try to fight it and might conceivably succeed. Then the precious thing would be lost."

"How long must the secret be kept?"

"Until the precious thing appears."

"How long?"

"Three hundred to five hundred years. The interval is uncertain. But there will be clear signs."

"And once the precious thing appears, the secret should be revealed?"

"Yes."

"Why?"

"Because people may misunderstand the significance of the precious thing. They will need to know why it appeared in order to know what to do with it."

I shifted in my seat. "How big is the secret?"

"Excuse me?"

"Is the secret the size of a deck of cards or the size of a super-tanker?"

"The secret is an explanation. Typed, it would fit on three or four pages."

"Who knows the secret now?"

"A dozen or so people in my organization. No one outside it."

"Can they be trusted?"

"Yes. We are confident they will not reveal it."
She seemed very certain. "That's not what we need help with."

"Can anyone in my organization learn the secret if we help you?"

"No. I suppose we can't conceal that there is a secret, but even you must not learn the details."

"So, to summarize, you need help keeping a modest bit of information secret for perhaps five hundred years and then revealing it to everyone. Is that a fair summary?"

"Fair enough."

I sat back and steepled my fingers. "We should be able to help you. It will take some time. You know our prices?"

She stood up. "Yes. As you know our resources. Thank you for this meeting." And with that, she walked out.

Higher education industry headed for trouble

A college education costs more than ever.

But the actual work of instruction is increasingly being done not by actual scholars but by adjunct instructors who are paid by the class at starvation rates.

And the student body at third-tier schools is often just plain not ready for traditional college classes.

So, colleges are charging ever more, and offering an increasingly debased product, to people who aren't actually able to receive it. Lovely industry, really.

I think the questions worth asking at this point are:
a) How long can this go on?
b) When it can't, what will happen?

Interestingly, some top companies have dropped their requirements for a college degree.

Squeezing Dune

Dune, the 1965 SF novel by Frank Herbert, has been adapted for the screen twice: first as a feature film by David Lynch in 1984 and later as a miniseries by John Harrison in 2000. A third effort is currently underway.

It's hard to squeeze Dune into a single movie. The book is a good 500 pages long, which fits better into three movies than into one. But if you are determined to fit it into a single movie, it should be possible. Here's one way:

ACT 1 (30 min) Start in Arrakeen, with the Atreides already in charge. Leto is the kind master, Paul his formidable young son. Their rivals are the Harkonnen, a nasty bunch (they abuse servants). Introduce Arrakis as the source of the Spice, possibly as part of Jessica's training of Paul in esoteric disciplines. The Fremen are mentioned, but as a minor impoverished rabble. Meanwhile, the Emperor, the Space Guild, and the Harkonnen meet and agree to replace the Atreides in order to maintain control of the Spice. The Harkonnen attack, Leto dies, and Paul and his mother flee to the desert. End with Paul looking back and vowing revenge.

ACT 2 (30 min) Paul and Jessica encounter the Fremen, and are taken in by them. Paul tests himself against young Fremen fighters and is impressed; they are very good, but he is better. He accompanies them on a raid against the Harkonnen, and is again impressed. He joins the Fremen, and becomes a worm-rider. He learns the true size of the Fremen from seeing one of their secret meetings. Jessica speaks of the Voice from the Outer World prophesy. Following Jessica's advice, Paul decides to lead the Fremen against the Harkonnen. Meanwhile, the Harkonnen are making a cruel mess of Arrakis.

ACT 3 (30 min) Paul campaigns to become the Fremen war-leader. This effort is shown as a montage of knife fighting and speeches before increasingly large gatherings. Harkonnen cruelties continue. The Emperor announces plans for a visit. Paul, Jessica and Stilgar begin training the troops, incorporating her and Paul's esoteric training. They wait for a sandstorm and attack in force with wormriders. Fight scenes between Fremen and Hakonnen. The fight goes to the Fremen. The Emperor flees in a ship. Paul addresses the crowds of Arrakeen from a balcony proclaiming a new day on Arrakis, with the natives in charge.

I think that could work. But some elements does get left out, given the compressed time-line: the spice as a mutagen, the nature of the Bene Gesserit and their goals, the Kwisatz Haderach, Paul's Harkonnen heritage, Paul's duel with Feyd-Rautha, and everything about Liet-Kynes.

Why might an atheist be trustworthy?

One of the charges that's sometimes leveled at atheists and agnostics is that they cannot be trusted to do the right thing because ultimately they don't have any reason to fear consequences when they do wrong. If no one's watching, the thinking goes, people steal. For believers there is always Someone watching but for unbelievers, there isn't. Therefore unbelievers cannot be trusted to do the right thing. This is a serious claim, and deserves a serious answer.

To begin with, is religious belief sufficient for moral behavior? Are all believers virtuous? No. Visit any of our prisons, or watch any program about them, and you'll see plenty of religion there. There's no shortage of God-talk and God-work in prison. If Jesus is everywhere, he seems to make a special point of visiting every jail-house in the land. So belief is not sufficient for moral behavior.

But if it isn't sufficient, might it be necessary? Does only belief motivate people to do the right thing? A believer who does something because he believes God will punish or reward him is motivated by feelings of fear (of consequences) or greed (for reward.) Greed and fear are certainly feelings that drive behavior. But they are not the only ones. Think of a mother comforting a crying child when she could just as well have let him cry himself out. She's doing it out of love. Or think of a soldier staying up through the night on guard when he knows damn well sarge is fast asleep in his bunk. He's doing it out of feelings of duty. Love and duty are feelings that can drive people to do the right thing, quite apart from fear and greed.

The feelings of duty can be specific or general. They can motivate the soldier to do his duty to his unit or corps, but they can also apply to larger duties, including the duty to do what is right in a broader sense. And it is these feeling of duty that an atheist can feel just as keenly as any believer, and which may lead him to do what is right.

Will they necessarily do so? No. Some real scoundrels don't have these feelings of duty. And even for people who do have them, they sometimes aren't sufficient to overcome temptation. But then the fear and greed that are supposed to motivate a believer aren't always sufficient either. Both believers and unbelievers may do wrong. My point is simply this: fear and greed are not the only feelings that may motivate people to do what is right. A feeling of duty may do so also. An atheist may feel a duty to do what he believes is right just as keenly as the most pious believer. And that's why an atheist may be trustworthy.

Why blunt criticism is often a bad idea

Sometimes it is tempting to just speak your mind . That's the honest thing to do, right? No reason to dissemble like a courtier; just say what you mean.

Unfortunately speaking plainly and directly in all cases runs into some pretty hard cultural boundaries. What you say and what people will interpret you as saying can be quite different. This will seem patently obvious to some people. I will explain it from first principles because it is not obvious to everyone.

The way the social protocol works around here (US/Canada) is that it is just fine to be directly critical of others if you are talking downward socially. Boss to subordinate, parent to child, teacher to student, it's just fine. But it's not really OK peer-to-peer or upward. And if you insist on being bluntly critical peer-to-peer or upward, what you are saying carries the additional meaning that the person is a real bozo or messed up really badly, so you are justified in talking down to them, as a sensible knowledgeable person dealing with a fuck-up.

That said, it is possible to convey criticism sideways or up, but it requires some social tap-dancing to emphasize that you are only telling the person they are mistaken, not that they are laughably hilariously wrong. The simplest of them is understatement. Compare "That doesn't look right to me. Could you check that it's doing what you expect?" to "No. That's wrong."

Next time you are tempted to be bluntly critical, keep this in mind. Your message may be taken as much more severe criticism than you intend.

Reviewed: Making the Corps

I just finished Making the Corps by Thomas Ricks. This is a book about the US Marine Corps. The first part shows the training of recruits in great detail, following one platoon through basic training at Parris Island. Later parts discuss the history and character of the Marines, their distinctive culture, and how that culture is transmitted, starting with basic training.

Finally the book discusses the growing distance between the armed services and the society they protect. In their eyes, the rest of us are fat, rude, greedy, and undisciplined slackers. We are also rather too liberal. Few Americans -- and very few members of the elite -- have actually worked in the military, which makes it hard for them to understand what the military does and why, exacerbating the problem.

Making the Corps is a great book. It has a generally positive view of the Marines, but isn't afraid to point out some of the rough bits, too.

How much does it cost to provide a college education?

The price of college-level study varies widely. At the top end, Harvard charges some $47,000 in tuition and fees per year. At the low end, CCNY charges in-state residents $3165 per semester. Subsidies and prestige pricing abound in higher education, obscuring actual costs dramatically. But how much does it actually cost to provide a college education? Let's try for a ballpark figure.

A college education usually consists of four years of study, two semesters per year, with maybe five courses per semester. That's a total of 40 courses.

The main expense of running a college is the professors who teach the courses. Suppose each professor teaches four course sections a year, two per semester. That will leave him or her time for a bit of scholarship during the school year, as befits a scholar who is not purely an instructor. The loaded cost of a professor might be $150,000 per year, including benefits. And each course section has 100 students on average. That comes out to $150,000/(4*100) = $375 per course for each student.

40 courses per student at $375 each comes to $15,000. But that's just the cost for the instructor. Colleges also have libraries, lecture halls, labs, office space, recruiting expenses, and a hierarchy of supervisors and deans, all of which come under the heading of "overhead". How high might those costs be?

Colleges typically take 50% off the top of every research grant awarded to their faculty members for overhead expenses, which suggests a useful and simple ratio of 1:1 in primary costs to overhead. So the original $15,000 becomes $30,000. Of course, that $30,000 excludes quite a bit: room and board, text-books, and general living expenses for those who aren't still living at home. But that's our estimate: a bare-bones college education costs $30,000.

Why the 2008 market crash had no satisfying conclusion

After the 2008 financial crash, the electorate was left in a distinctly unpalatable position. The government had been forced to step in to rescue several major institutions, leaving the taxpayers with a hefty bill, but nearly all of the key players in the market were left standing. Things had gone wrong but no one, it seemed, could be blamed and punished.

In my view, the resolution was so unsatisfying because the blame for the crisis stretched far. There was no single institution or even sector that single-handedly caused the problem; many, many people and institutions played a part.

• Ordinary homeowners sought mortgages they couldn't really afford.
• Mortgage brokers sold mortgages to people with very poor finances, who probably should not have been in the mortgage market at all.
• Originate-and-sell lenders didn't particularly scrutinize the mortgages they were issuing, because they could always sell them on.
• Investment banks sliced and diced pools of increasingly crappy mortgages into ever more baroque bonds, without a proper analysis of systemic risk.
• Bond rating agencies went along with what the banks were building, happy to rate them AAA as long as they collected their fees. 
• Institutional investors bought the baroque bonds based on ratings alone, without scrutinizing the internals.
• The US Fed kept interest rates low for a long, long time, exacerbating the housing bubble that drove the system.

Who was to blame? All of them? None of them? Probably all of them, to some extent. But with no clear villain, there was no one to punish, and no sector which could be targeted for clear reforms. Hence the dissatisfaction.

Should you use Haskell for a new project?

What are the pros and cons of trying to do a new project in Haskell, as opposed to some more mundane imperative language like Python or Java? Let's assume for the sake of argument that you are doing this for an employer, rather than for yourself as a side project. And let's also assume that you have a bit of Haskell experience, but like most programmers most of your work has been in mundane imperative languages.

To begin with, there are a couple of concerns that might kill an attempt to use Haskell.

1. You might not be able to get management approval. Haskell is an obscure language with a reputation for difficulty. The bosses might well say no. 
2. You might need to work with an existing codebase in something like Java or Python. Not possible from Haskell.

If those don't kill the project, there's the cost-benefit tradeoff. On the benefits side:

1. Haskell code is very concise. (definitely) And greater concision means faster development. (possibly) 
2. Haskell code is less likely to contain errors, because of very strict typing. (probably) 
3. Haskell coders are disproportionately capable, because the language is obscure and difficult. (probably)

On the costs side:

1. You have much more experience in other languages. It would take at least a year, maybe two before you'd be up to pro standards in Haskell. 
2. The community of Haskell programmers is small. It might be difficult to hire anyone if the project grew; you might be faced with training someone from scratch, not just in a new language, but in a new programming paradigm.

So, this isn't anything like a slam dunk. Quite the opposite, actually. There are a couple of issues that might nix the project right up front, and then a daunting cost-benefit calculation.

Reviewed: Masterpieces: The Best Science Fiction of the Twentieth Century

Masterpieces: The Best Science Fiction of the Twentieth Century is an anthology of science fiction short stories compiled by Orson Scott Card, one of the giants of the genre. He's the man who wrote Ender's Game and its many many sequels.

The anthology collects 27 short stories spanning much of the twentieth century, from 1936 to 1995. The best of the stories are very good indeed. "The Road Not Taken" is an intriguing what-if by Harry Turtledove. "'Repent, Harlequin!' Said the Ticktockman" has an appealing rebelliousness and Harlan Ellison's awesome prose. "Dogfight" by Gibson and Swanwick is gritty and fierce. And "The Ones Who Walk Away from Omelas" by Le Guin poses a really hard moral question.

But, but, but. There are too many stories that are just sort of ok-ish -- 16 out of 27 by my count. If you're stepping up to the plate with "Masterpiece" stenciled on your bat, you should be whiffing less often than that.

The most puzzling inclusion is "'All You Zombies--" by Robert Heinlein. It makes perfect sense to include something by the grand old man of the genre. But why pick this odd little time-travel story when there's such a wealth of great alternatives? Card could have chosen "The Logic of Empire", "The Roads Must Roll", "The Man Who Sold the Moon", or one of at least a dozen other first-class stories instead.

Overall then, this is a solid collection, with some very good stories by big names in the genre. If only there were more of them.

3/4

Where to find the best science fiction short stories

I recently went looking for some collections of really top classic science fiction stories. Here's what I found.

If you're looking for a single volume, you probably can't do better than Masterpieces: The Best Science Fiction of the 20th Century. The collection covers a broad span of time, includes really big names, and the anthologist is Orson Scott Card, a highly respected author in the genre.

Beyond that, the ways part. The SFWA, the professional association of SF writers, got together in the early seventies and assembled The Science Fiction Hall of Fame, a collection of science fiction masterworks in three volumes, the first covering short stories and the other two covering novellas.

For more recent work, you might try The Best of the Best: 20 Years of The Year's Best Science Fiction by Gardener Dozois. Dozois was for a long time the editor of Asimov's, the top forum of the field for short fiction, as well as the editor of The Year's Best Science Fiction, an annual anthology of short stories. He has collected some of his top choices of more recent material in two volumes, one for short stories and the other for novellas.

And for even more recent material, there is Twenty-First Century Science Fiction, published in 2013.

Grade 12 Math: Something Other than Calculus

In the US and Canada, the mathematics taught in the senior year of high school is typically linear algebra (matrices, determinants, inversion, etc.) and calculus, meaning single-variable differentiation and integration.

I'm not sure this is wise, for two reasons. First, linear algebra and calculus are remarkably useless for people who aren't headed for STEM occupations. They are vital for some engineers and scientists, who use them all the time, but don't get much use outside these disciplines. I work in a pretty technical domain -- I'm a software developer -- but I've used calculus exactly once outside of schoolwork and linear algebra never at all.

Second, universities usually insist of teaching these two topics anyway. In STEM programs, they are typically covered in mandatory courses in the first year. Students whose high schools did a good job get to coast for half a year, while the less fortunate catch up, which is just wasted time.

If we eliminated linear algebra and calculus from high school curricula, leaving them to those who are headed for professions where they are actually useful, we could replace these courses with some other forms of mathematics. There are plenty of choices: abstract algebra, topology, geometry, combinatorics, and the list goes on.

Of these, formal logic and statistics are particularly promising. Formal logic attempts to rigorously determine the truth value of claims, while statistics handles approximations and drawing inferences from observed behavior. As such, they are particularly useful for sorting through the mass of questionable claims we all face in an information society, to find the truth among the wishful thinking, half-truths, and outright lies. This makes them valuable aids to good citizenship, which is something we should be aiming everyone towards. And as actual domains of mathematics, they can be made as rigorous and challenging as the less useful courses they would be replacing.

Four Solutions to the JavaScript Problem

If you have worked with JavaScript, you are surely familiar with a long litany of complaints about the language. Some developers are frustrated by its very loose, dynamic nature, while others point to odd behaviors such as hoisting and conversion-on-equality as big problems. (More here.)

I don't want to belabor the problem itself; plenty has been written about that already. Instead, let's consider the solutions. There are essentially four types, at increasing distances from plain JavaScript.

Use JavaScript Carefully: Program in JavaScript directly, but avoid or work around the sharp edges of the language, such as weak typing and the lack of a module system. In effect, you program in a restricted sub-language of JavaScript. This is what Douglas Crockford and Stoyan Stefanov advocate in their books. Crockford goes even further with JSLint, a verifier that strictly enforces the use of his dialect of JavaScript. 

Add Annotations: Add hints about the programmer's intent to the code, typically with type annotations. This requires a compiler to process the language, but both the input and output languages of the compiler are JavaScript -- perhaps even identical to the input if the annotations read as comments in JavaScript itself. This is much of what TypeScript and Closure try to do, although their compilers go a bit farther, adding things like a module system.

Use a Neighboring Language: Program in a fundamentally different language than JavaScript. But keep the data model and functionality close enough that translating to JavaScript isn't too hard. With a small semantic distance, the output is recognizably similar to the original code, making it useful for debugging. CoffeeScript and PureScript follow this model. 

Use Something Else Entirely: It is possible to compile virtually any programming language to JavaScript, although more distant languages require a lot of extra JavaScript code output to bridge the semantic gap. And of course more code means more execution time. Kiss source-level debugging goodbye too. GWT and Haste do it this way.

Where you want to be on this continuum depends on how troublesome you find JavaScript and how much control you are willing to hand over to a compiler and runtime system.

Steve Yegge on Google's Engineering Culture

The ever-voluminous Steve Yegge wrote an interesting post back in 2012 about liberal and conservative elements in programming. By "conservative", he means practices that emphasize safety and rigor; "liberal", conversely, emphasize freedom and flexibility. He then used these labels to categorize programming languages, features, development practices, and cultures.

Here's what he had to say about Google's engineering culture:

Google -- Diagnosis: Conservative. They began life as slightly liberal and have grown more conservative ever since. Google was only software-liberal in the very very early days, back when the search engine itself was written in Python. As they grew, they quickly acquired a software conservatism driven entirely by the engineers themselves.  Manifestos were written about the dangers of using multiple languages, and strict style guides were put in place to severely limit "risky" or "hard to read" language features of the few languages they did allow.  ...  In internal surveys, Google engineers commonly cite bureaucracy, churn and complexity as core obstacles to feature advancement and rapid launches.  Google has made serious attempts on several occasions to reduce this bureaucracy, but they always get pushback from -- surprise -- the engineers themselves, who have grown so staunchly conservative that they actively (and even more often, passively) resist the introduction of more flexible stacks and technologies.  ...

Having seen Google from the inside, I can say Steve nailed it. I had expected Google to be sort of an over-grown startup, a really intense place where you had to work extremely hard, but where you had a lot of freedom to do things however you wanted, as long as it worked. What I found was something quite different: a culture that was incredibly strict[1], top-down[2], and bureaucratic[3].

If Steve is right, what I expected to find at Google is actually found at Amazon and Facebook.

[1]: Check out the C++ style guide (click "Toggle all summaries" to see the document in its full form). Then consider that the code review culture routinely enforces all of it. I remember having check-ins rejected for spaces at the ends of lines, for example.

[2]: I still think I should have quit or transferred the day my lead rejected the fifth rewrite of a design document.

[3]: The most prominent manifestation of Google's bureaucratic nature is the "promo packet", an elaborate document used to apply for promotion. It is typically at least a dozen pages long, with a long series of testimonials by the applicant's bosses and peers singing his praises for the promotion committee. It's like going up for tenure or a partnership, but it's done for every step up, not just once in your career.

The Joy of the Trailing Edge

Today I signed up for Skype. For the first time.

I installed a webcam, downloaded the software, signed up for the service, and it all went without a hitch. No fiddling with drivers, no screwing around with setup files, no searching through FAQs. It all Just Worked.

Plus everyone I might want to talk to is already there. Including mom.

The joy of the trailing edge.

Simple Fixes for Improved Mobile Web Browsing

I recently started reading Hacker News, and discovered to my frustration that a) the main site is not really useable from a mobile phone and b) they don't have a mobile version. I played around with the HTML, and found that it wouldn't be hard to improve the mobile experience significantly. These tweaks to the HTML make a real difference by letting the content wrap to the screen:

• Add <meta name="viewport" content="width=device-width, initial-scale=1.0"> to the <head> element.
• Remove the width="100%" attribute from the table (and maybe the width="85%" from the sub-table, too).

 As it turns out, several of the sites I visit regularly are hard to use on mobile devices:

• http://www.rifters.com/crawl/ 
• http://www.antipope.org/charlie/ 
• http://forums.sjgames.com/

But every problem is an opportunity, right? It should be possible to create a system that uses various transformations to reformat web pages to make them more useable on mobile browsers. It might boost text sizes, relax table specs, and push sidebars to the end of the page, for example. The result wouldn't match custom designs, of course, but might get the pages from bad to useable, particularly if the user has some input into what transformations are applied.

I wonder what tweaks would help those other sites ...

Building a Compiler, Briefly

During the last meeting of the Toronto Haskell group, we briefly discussed the question of how to build a compiler, and in particular the problem that building a real one is a daunting task.

I've done a bit of digging into various approaches and guides. There are a lot of them, because every respectable CS curriculum needs a compilers course.

The one I find most intriguing is by Niklaus Wirth, the guy behind Pascal and Modula-2. In his "Compiler Construction" PDF, he guides you through building a compiler as a series of exercises in a bit over a hundred pages. Given that other texts run well past 500 pages, a tractable treatment is quite a relief. I get the impression you could work through the guide in 2-3 of weeks full time or perhaps a quarter as a side project.

That said, Wirth does make some simplifications for pedagogical purposes. The source language is Oberon-0, which is a pared-down version of Oberon, so almost but not quite a real language. The target architecture is a simplified RISC instruction set, with an interpreter that fits in a page. You build the compiler from scratch, rather than using tooling like ANTRL, parsec, or LLVM, which you'd presumably leverage in an actual implementation. And the treatment of code optimization is brief. So, choices were definitely made.

But all in all it is possible to make building a simple compiler the work of a couple of weeks rather than a months-long slog.

Joel Spolsky on Language Wars

Joel Spolsky is the CEO of Fog Creek, a software company in New York. He also blogs about software development on his website, Joel on Software. In one of his older posts, he addresses the question of what language and framework is best for web development, and drops some giant blocks of stone-cold sense right in front of you. 

Some choice quotes, condensed from the article:

Which Web Server (Apache, IIS, or something else) should we use and why? 

People all over the world are constantly building web applications using .NET, using Java, and using PHP all the time. None of them are failing because of the choice of technology.   

All of these environments are large and complex and you really need at least one architect with serious experience developing for the one you choose, because otherwise you'll do things wrong and wind up with messy code that needs to be restructured.

How do you decide between C#, Java, PHP, and Python? The only real difference is which one you know better. If you have a serious Java guru on your team who has build several large systems successfully with Java, you're going to be a hell of a lot more successful with Java than with C#, not because Java is a better language (it's not, but the differences are too minor to matter) but because he knows it better. 

Yes. Absolutely yes. There are several different solutions out there, all of them work, and the best choice is likely to depend on something other than a theoretical determination of which one would be best in Plato's world of pure forms, for any number of reasons:

1. You probably aren't building the system from nothing. You are starting with an existing system, and adding on to it. Using whatever language or framework the system is already built from is a big advantage, because the new stuff needs to work with the old.
2. You know something but not everything. A Java-based solution would need something very special indeed to do better than a Python-based solution, if you already know Python backwards and forwards but you've never touched Java. 
3. There is already a standard solution. Someone has already designated a language or framework as standard in your organization, hopefully after carefully weighing costs and benefits, but maybe not. In any case, using anything else would require an arduous process of argument and justification, and every day you spend on the fight is a day you could have spent designing and building your system.

Go read the article. Really. Giant blocks of stone-cold sense. Here's that link again.