The Beauty of Lisp

• Programming languages are about turning your ideas into a form a computer can understand.
  
  • The ideas must be mathematically defined, but beyond that they can be arbitrary (Turing equivalence).
• Most good ideas need to be reused.
• Programming involves defining ideas and reducing them to "words", and then referencing those words whenever the idea needs to be reused.
  
  • In this way, programs can be built in layers, one on top of the other, where each layer is abstracted away to make room for the next one.
    
    • A good language has semantics that let you easily label common patterns of ideas (such as functions, data types, and object classes).
    • A compiler turns source code from that language into machine code.
• Some ideas can't be reduced down to a single word in certain languages, because they don't follow the patterns the language designers anticipated, even though the ideas themselves are mathematically defined.
• As a result, the programmer has to "preprocess" his ideas into source code — a mechanical task but one necessitated by the lack of expressiveness of the language. In essence, the programmer is doing the work of the compiler.
• Therefore, what's needed is the ability to define arbitrary patterns of ideas, not just arbitrary ideas.
  
  • The only way to do this, is to be able to define the patterns in the programming language itself — to write programs which can manipulate programs.
    
    • That is what Lisp is.

The true price of software?

Robert Lefkowitz has written an interesting analysis of the true value of computer products, by breaking down those products into their components, much as an economist breaks down financial instruments into their components, or a theoretical physicist breaks down particles into their elementary primitives. He identifies several services: repairs, maintenance, and the option to upgrade to future versions, that in his view constitute a large percentage of the value of the product, and concludes that, in the case of software, the underlying software itself has essentially very little value compared to the rights to support and upgrades.

I believe his analysis is seriously flawed, and not simply because I dispute his assumptions as to the volatility of future support/upgrade pricing. Let's go through some of his examples:

A 17-inch iMac was $1,499 at the Apple store the last time I checked. You can purchase an extended warranty, AppleCare, for $169. That warranty is for years two and three; year one is included. AppleCare also includes extended telephone support, but I'm going to ignore that for now to simplify things.

After a quick trip to Wikipedia's page on failure rates (leavened with anecdotal rumors), it is not unreasonable to suppose that computers experience more failures in their first year than in the subsequent two years. The overall failure rate for computers runs about 15 percent--Macs do better than average. Still, it is not unreasonable to suggest that the curve looks roughly something like 8 percent failure in the first year, 4 percent in the second, and 2 percent in the third. That means the first-year warranty is worth about $225. So really, that 17-inch iMac costs $1,274 for the computer and $225 for the first-year warranty.

He apparently derives this $225 figure for the value of year one of the warranty by multiplying 15% times the cost of the iMac, but there are two problems with that. First, and trivially, he assumes the iMac's failure rate in the first year is only 8%, so the value of AppleCare during that time should be 8% of the value of the underlying hardware, not 15%. But second, even if the first-year failure rate is 15%, it should be 15% of the value of the hardware, not 15% of the value of the hardware plus the value of the AppleCare. In other words:

AppleCare + Hardware = $1,499
AppleCare = 0.15 x Hardware
Hardware = $1,499 / 1.15 = 0.87 x $1,499
AppleCare = 0.13 x $1,499 = $196

Not too far off from his value of $225 (assuming 15%, not 8%, failure), but this has exposed a basic flaw in his reasoning which will become much more relevant when we get into the discussion of software. Namely, he bases his reasoning on the very assumptions which his reasoning disproves. In this case, he fails to price the value of the AppleCare relative to his logic's implied value of the hardware — rather, he values the AppleCare relative to the same naïve valuation of hardware that he has just shown is faulty.

Now, let's move onto software. Here's his thought experiment:

Let's normalize the values to a $100 software license and say that a one-year option has a $20 underlying price; a year of maintenance is 20 percent of the license, so we'll assume it's worth $20 today. The strike price (what you can buy it for in a year) is also $20--a 5 percent risk-free rate. With all of those inputs, the value of that option is about $2.85. That is to say, for $2.85 you can lock in the price of the maintenance contract so that one year from now, you'll have the right to buy it for $20.

The right to buy the same maintenance for $20 two years from now is about $4.25; three years is $5.35; four years is $6.35. That takes us five years out. Assuming that you've locked in the maintenance over the five years to 20 percent of the purchase price, that set of options is worth $2.85 + $4.25 + $5.35 + $6.35 = $18.80. Five years is not an unreasonable horizon for enterprise software.

But wait, there's more.

Another option included in the license price is the option to upgrade to future versions at some price that will be less than the regular price. Right? That's clearly an option. The same informal poll of enterprise software users asked what they would pay for software when they didn't have the option to upgrade to the next release. The strike price is less standard than regular maintenance. (If you think about it, new versions perform "maintenance" by adding features as opposed to fixing bugs.) Now we're buying an option to upgrade in five years as part of this license.

The underlying price--the price you'd have to pay if you didn't have an option--we'll leave at $100. The next version will be priced the same as this one. Because you're upgrading, you have an option with a strike price of, let's say, $50. That is, you'll be able to upgrade to the new version for only $50. A five-year option for a $100 underlying price with a strike price of $50 and a volatility of 30 percent (with a 5 percent risk-free rate) is about $62.50.

Of course, most software offers new releases more frequently than once every five years--but enterprises don't like to upgrade very often and usually plan on skipping every other release in order to avoid upgrading too often. Then it would be two options. Much like adding additional planetary bodies to a problem in gravitational dynamics, the complexity mounts rapidly. I'm trying to keep it simple. (I suggest follow-up research problems for interested students.)

At this point, the astute reader will have noticed that the sum of the value of the option for the upgrades plus the options for the maintenance is $18.80 + $62.50 = $81.30. That is to say, our $100 software license consists of $18.70 for the value of the actual software and $81.30 for options on future maintenance and enhancements.

But wait a minute! If the right to upgrade is already included in the first purchase, then what is the upgrade itself getting you? Only the software plus a new maintenance contract, which is worth just $37.50! Sure, you now have the right to upgrade to future versions, but you already had that right. So if his conclusion is correct, and the underlying software+support is only worth $37.50, then in fact the right to upgrade at $50 is worth almost nothing — it's a call option that is way out of the money!

Lefkowitz's approach has been exposed as an accidental pyramid scheme, and the foundation has just come crashing down!

Now, I am not saying the right to upgrade is worth almost nothing. That would be making the same fallacy that Lefkowitz makes. I am saying that his logic is self-contradictory, and that a proper analysis will find an equilibrium price for both the right to upgrade and the immediate product, rather than consisting of one "iteration" of analysis on assumptions that the very analysis has shown to be wrong.

There's another, more abstract problem with his reasoning: he's attempting to seperate inherently complementary products, and arguing that the value of one is virtually nil without the other. But that process works in reverse, too. What's the right to maintenance worth without the actual software? What's a car with no steering wheel good for?

The answer is, a car with no steering wheel is good for making a car, once you've got the steering wheel. But this doesn't mean the steering wheel is the majority of the value of the car. It means the car is worth X, and the individual components are worth whatever the supply and demand situation says they're worth. That calculation is substantially more complex than just subtracting the utility of a car with no steering wheel from X. You must consider factors such as the consumer's desperation, the cost to the supplier, and the level of competition and commoditization in the marketplace.

It may be that a good business model can be built on cheap or free software, but Lefkowitz's argument is surely not why.

Why RSS feeds don't work for Reddit

(For those who don't know what Reddit is, it's a content aggregator, similar to Slashdot and Digg, but much better, and with personalized recommendations.)

The bulk of RSS-syndicated content — blogs, news sites, and most content aggregators like Slashdot — are essentially a chronological stream of data. New stories/links appear one by one, and there's really no other organization to it than that. There's no attempt to rank the entries by how relevant they are. They either are or aren't on the "front page" (or whatever page you're looking at), and then they're just sorted chronologically. Thus, RSS feeds, which basically assume such a sequential content stream, are perfectly suited for these sites.

Digg is an exception to this. With Digg, article A can be placed higher up than article B, even though it was submitted several hours earlier, simply because it's received more diggs. However, there is still a strong bias towards chronological arrangement. It's extremely rare for an article to last a full day on the front page, before other articles are considered more "relevant".

Yet, with Reddit, entries frequently remain on the front page for one day or more, as a nagging reminder that "hey, this is really popular, and you should read this if you haven't yet." For example, I haven't read This Essay Breaks the Law yet, even though it's been in the top three on my recommended page for over a day. Normally I wouldn't have read it, since there are plenty of contributions to the intellectual property debate and I usually don't feel they offer something new. But now it's on my to-do list, because so many people have read it and liked it.

Sometimes I'll go two or three days without having time to read stuff. When I check out Reddit after that period, I can immediately see the absolute most relevant stuff during that period, rather than artificially forcing myself to sludge through the same number of articles I would have read had I actually had the time those couple of days, as you might do if you're trying to keep track of the content on an RSS feed.

Also, sometimes I'll have a lot of free time or just be generally interested in checking new stuff out, and then I can browse through all the links in the first page or two of reddit recommendations. And, as always, they're organized as I'd want them to be: weighted by both relevance and timeliness.

What it comes down to is that Reddit's content isn't a "stream" so much as a constantly-shuffled pile of stuff, arranged by that weighting of relevance and timeliness I just mentioned. My Reddit RSS feed simply can't have the real-time organization that the Reddit site has, without being a mirror of it. And, in that case, what's the point of RSS?

Quantum-encrypted network

First quantumly-encrypted wireless network installed in Massachusetts.

Proof that we need WebHashCash

That's a verification image I encounted on Yahoo today. I was able to decipher it ("pl7p"), but it's clear the arms race is being won by the spammers.

End the arms race. Use WebHashCash.

Dvorak Progress

I just hit 82 words per minute for an extended five minute period for the first time today, typing in Dvorak. I'm now pretty comfortable with the layout, and I don't think learning it has substantially harmed my QWERTY skills.

I tend to switch back and forth between the layouts a couple times an hour, depending on my mood. Dvorak is better for lazier typing, as it requires much less hand motion and overall energy. QWERTY is still better for having very intensive conversations or writing long compositions, since it requires slightly less mental engagement for me, and doesn't slow me down as much.

My hands sometimes hurt after extended Dvorak typing just as they sometimes do after extended QWERTY typing. So another advantage of frequently switching back and forth between the two layouts is that they use different kinds of motions, giving certain muscles a rest for a while and then I can switch back when I feel comfortable.

QWERTY is also better for programming, as programming really isn't about writing long sequences of English text, so many of the advantages of Dvorak disappear. And many useful characters are in optimal places for a QWERTY keyboard.

On the whole, I'm definitely glad I've learned Dvorak. Hopefully I can get above 100 words per minute in the next several months.

(This post typed in Dvorak.)

WebHashcash — The End of Comment Spam

I am pleased today to announce WebHashcash (based on Hashcash by Adam Back), a fundamentally new mechanism for preventing spam on web sites of all kinds: blogs, discussion forums, wikis, and more. It guards against automated content posting, fake user registration, and ballot stuffing.

Astute visitors to my blog may have noticed for the past month a subtle little doodad on the comment submission form. That's WebHashcash functioning. While it's running, your CPU is maxed out, and when it's done (after a couple seconds), it will have generated an invisible "stamp" which it attached to your message, proving that your computer invested effort in generating the stamp. It is computationally infeasible to fake such a stamp — its presence proves an investment in CPU time.

The basic idea is that spammers only spam because it's so cheap to do so. The value of a spam posting to a spammer is far below the value of a legitimate posting to its author. Thus, it's possible to set a "postage" that's well below the value of legitimate postings, but above the value of spam. You therefore destroy the economics of spamming by requiring a negligible but real expenditure for each message.

This is qualitatively different from every other anti-spam mechanism in existence today. The basic concept behind virtually all other systems is to create some sort of "Turing test" which humans can pass but computers fail at. This typically involves optical character recognition, or responses to challenge questions. The problem with this approach is that it entails a constant arms race — as spambots become more intelligent, the anti-spam programs must be improved as well. There is no clear winner, and the trend seems to be favoring the spammers.

Enter WebHashcash. Unlike traditional anti-spam mechanisms, no attempt is made to actually prevent spammers from posting. But they have to pay the cost. And no spammer will, because it would bankrupt them to do so.

Since I introduced WebHashcash on my blog one month ago, it has blocked 670 out of 734 spam comments. The remaining 64 comments were exceptionally easy to delete as they all came from the same source. Furthermore, the spammers spent several minutes of CPU time in total in order to post the comments. If WebHashcash were in widespread use, spammers would quickly run out of CPU power attempting to post on WebHashcash-powered sites.

Please check out the WebHashcash web site for more information and for instructions on how to install it onto your web site.

I'm also interested in hearing your thoughts. Please post them here!

Dvorak Typing

I'm a pretty fast typist, reaching speeds of 125 words per minute for short bursts, and sustained speeds of around 100 words per minute. But over the past several months, I've been experiencing a bit of pain in my hands and fingers.

So I explored the possibility of dictation software, such as IBM's ViaVoice and MacSpeech's iListen. I almost purchased iListen, but the expense ($150 with microphone) and reported inaccuracies caused me to hold off.

Instead, I decided to rearrange my keys and to try to learn the Dvorak keyboard layout, which is purported to be much more efficient and less of a strain on the wrists than QWERTY is, though market forces have kept QWERTY in widespread usage. Here's a good page with a seemingly rational point of view on the matter, and here's Wikipedia's informative entry.

So far, I've spent about 12 hours of typing time on Dvorak, and I can touch-type at 20-25 words per minute. That's a very frustrating speed, but I can switch back to QWERTY when I need to type fast, and I don't have to rearrange my keys because I don't need to see them.

I'm using Ten Thumbs Typing Tutor to learn, mainly because it was the first one I found. It generates some humorous random phrases for you to type (inspiring me to rewrite Vial, a nonsensical chatbot I wrote). For example:

If you want to start learning, most would recommend Mavis Beacon Teaches Typing.

I don't blame most people for using QWERTY to type, and I don't blame any computer or keyboard manufacturers for selling what people want, but for someone like me who uses a computer over ten hours each day, using the most efficient input method possible is vital. I expect it to take a couple weeks to reach a bearable speed, and hopefully from there I'll be able to quickly match my QWERTY speed, with less hand strain to boot.

I'll let you know whether it was worth it.

(This post typed in Dvorak.)

Study finds dogs can smell cancer

Pretty incredible:

Idea Futures Exchanges

There's an interesting and novel concept called an idea futures exchange. It works just like a stock or commodities exchange, in that market prices are set by a two-sided auction of bids and asks. The difference is that, instead of trading shares of a company, or quantities of a good, you're buying contracts which become worth something if a predefined event happens. So, for example, if you believe that George Bush is going to be reelected, you could currently purchase shares of such a contract at $0.53, and if he is in fact reelected in November, the shares become worth $1.00.

At first glance this may look like a glorified form of gambling. But because of the significance of the events being traded, these markets actually provide a very valuable service: information. They aggregate all the knowledge contained by their traders, and set prices that correspond, roughly, with the perceived probability of the event actually happening. Thus, the market currently believes that George Bush has 53% odds of being reelected.

Now, there are some caveats. Besides the fact that the market could be plain wrong in its odds estimation, there are also reasons why the market price wouldn't be the same as the market's perceived odds. There are several factors which could cause this.

One of them is risk management: a good analogy to this phenomenon is insurance. Insurance companies lose money when their customers suffer setbacks such as injury or property damage. Like any profitable business, they make money because they charge enough in premiums so that their revenues exceed their expected payouts. This means that, from a customer's point of view, you can expect that, over your lifetime, buying insurance will probably not pay off. Yet, most people buy insurance, as they should, because the value of insurance in risk protection far outweighs the cost. So, in the case of an idea futures market, a particular contract may have what's called hedge value. If investors perceive the election of George Bush to be bad for the economy, they might buy shares of George Bush as insurance, at prices higher than the actual odds of him being reelected. Similarly, if they think George Bush would be good for the economy, then the market price of Bush's reelection will be below the actual odds. It can be hard to separate this effect from the market price, to derive the implied probability.

Another factor is contract expiration date. Currently, you can purchase a five-year U.S. treasury bond (considered one of the safest investments possible), at a rate which would give you a virtually guaranteed return of 3.64% per year, which would compound to a 19.57% return over five years. The only way you would not receive this return is if the U.S. Government defaults on its loans, and if it does that, you've got bigger problems to worry about. Therefore, no one will pay more than $0.84 ($1.00 / 1.1957), today, to receive $1.00 in five years. In other words, even an idea futures contract which is guaranteed to be worth something in five years, will trade at a sharp discount today. Unlike the first factor, however, this effect can be relatively easily separated from the market price, because U.S. treasury bonds serve as a base of reference.

Despite these and other caveats, the market price of an idea futures contract still provides a very valuable, and usually very accurate odds prediction of the underlying event it is based on. And, as I said, though the market is sometimes just plain wrong about the odds, it rewards traders who can detect this and drive the price in the correct direction, so over time the market improves its predictive ability by giving the best traders more money to trade in the market, and therefore to effect the prices.

I trade on three idea futures exchanges. The first, the Foresight Exchange, uses fake money. The other two, the Iowa Electronic Markets, as well as Intrade, involve real cash, and I've done fairly well, turning a $250 initial investment into $780, in about seven months.