EnTech

Which operation def macro should I use in Cascalog?

2010-12-27T10:19:00.000-08:00

I've been using the Cascalog query language for Hadoop map/reduce jobs for a while. The learning curve involves coming up to speed on the rich set of powerful operation creation macros (essentially, they are various techniques for creating user defined functions). As such, I've put together the following chart to help describe the different operation types in Cascalog. The idea is to provide some guidance on which type is appropriate for a particular job; along with examples and notes on usage and performance.

Cascalog jobs/queries are written in Clojure (a lisp like language that runs on the Java VM), so all examples below are Clojure code.

Let me know if you see any mistakes, or if you have suggestions for further details that could be added to make the chart more useful.

For reference, this thread also has a description of the def macros from Nathan, and more examples.

How to unit test Apache Hive scripts, including those made for EMR

2010-10-09T17:08:00.000-07:00

Apache hive has proven to be a useful tool for writing Hadoop Map/Reduce queries. To do something a little more complex, requires some string manipulation. If you're using Amazon's EMR version of Hive, it includes the ability to provide values for variables at run time. But what if you aren't using EMR, or you are, but you want to test those scripts locally?

GHive is a small Groovy wrapper around Hive. It provides a few basic functionalities. It can be particularly useful in the cases mentioned above. I wrote this a couple of months ago, and I'm releasing it as open source now. The code is at http://github.com/mlimotte/GHive.

FEATURES

Variable Interpolation: The script can include variables of the form ${name} which are replaced by values from the vars Map. This is identical to the Amazon EMR extended functionality. It's a erally useful feature, and makes sense that it should be available without EMR. Ideally, it will be added to mainstream Hive, eventually.

Comments: If you have a long query, it is useful to include comments, just like you would in a long SQL query. With Hive, you can do that for a script run with "hive -f", but not for queries that are executed through JDBC. With GHive, you can include comments with a -- at the beginning of a line or whitespace followed by --. You can not put a comment at the end of a line. That's because I didn't want to interfere with scripts that might include a string with -- in them. For example,

WHERE my_field = '--'

...would be a problem. It would be nice to fix this with a smarter regex or parser, but it wasn't worth the trouble for me.

Dump processed script to a file: It can be useful to save a copy of the script, after the GHive pre-processor has worked on it. A simple use case is a complicated hive job that runs once a day. If you dump the script to a log directory, you can refer to it later if there are errors and even run the individual commands interactively through the Hive shell. There is a simple GHive API call to do this (see below).

Multiple commands through JDBC: The commands are sent through the Hive JDBC driver. If you have a set of distinct queries in a text file and try to feed them all to the JDBC driver at once, you'll notice that JDBC only accepts one command at a time. So GHive separates the script into multiple commands, which are fed to JDBC serially. Commands are terminated by ";" or EOF.

BUILDING

Install Apache Ant (I used version 1.7.1)
Adjust any paths in build.xml
Run:
ant jar
The resulting jar will be at ./ghive.jar

USAGE

Example 1 (simple script)

Create a hive script with variables and comments. For example, create a file "hive/simple.ghive":

-- My simple hive script
-- For simplicity, I'm assuming the tables already exist

ADD JAR ${HIVE_LIB}/hive_contrib.jar;

ALTER TABLE user_ex ADD PARTITION (dt='${DATE}');

INSERT OVERWRITE TABLE tmp
SELECT username,
-- first_name,
-- last_name,
email,
phone
FROM user_ex;

In your Groovy Code

GHive ghive = GHive.instance()

// vars are a Map, the keys are case-sensitive. Remember, in
// Groovy, symbols used as keys in a map don't need to be quoted.
// I.e. [ FOO : 'foo' ] is equivalent to [ 'FOO' : foo ]
def vars = [
                   HIVE_LIB : '/usr/lib/hive/lib',
                   DATE : '2010-07-02'
                   ]

// The use of dumpScript is optional, and just writes a copy of the GHive
// processed hive commands to disk. The resulting file could be fed
// directly to the hive shell via the -f flag.
ghive.dumpScript("hive/simple.ghive",vars,"output/simple.hive")
ghive.executeScript("hive/simple.ghive",vars)

Example 2

Run some query on each for a list of names and process the result in hive. For example, a script "hive/getdata.ghive":

-- A user may be in multiple groups
SELECT username, group
FROM user_group
WHERE username = ${USERNAME}

And groovy code:

GHive ghive = GHive.instance()
def usernames = [ 'gilbert', 'brook', 'xtreme' ]
usernames.each { username ->
   ghive.eachRow("hive/getdata.ghive", [ USERNAME : username ]) { rs ->
   def group = rs.getString(2) // Like standard java sql, a 1 based index number
     println "$username is a member of $group"
   }
}

UNIT TESTING

See "test/ghive/TestSimple.groovy". The main testcase method from that example is here:

@Test
public void testSimple() {

   // the q path is relative to the classpath
   def q = "simple.hive"

   def queries = ghive.parseScript (q, [ STORAGE_TYPE: 'TEXTFILE' ])

   ghive.execute(queries[0]) // create table
   ghive.execute(queries[1]) // load data
   def result = ghive.executeAndGetList(
                      "select id, value, amt from simple",
                          [ 'id', 'value', 'amt' ])

assertEquals(3,result.size())

def expected = [
        [ id:'1', value:'line1', amt:'0.2' ],
        [ id:'100', value:'line2', amt:'0.3' ],
        [ id:'50', value:'line3', amt:'0.4' ]]

assertEquals(expected,result)

}

THOUGHTS ON THE DSL

This is a simple, external DSL. Conceivably, I could use a full parser instead of the simple REGEXs and expand this into a full DSL with conditionals and loops and so on. But, as Example 2 shows, you can just use groovy to do this.

If that's a typical use case, it might make more sense to create an internal DSL.

How to move files from HDFS into Amazon S3 - A shell script wrapper for distcp

2010-09-22T17:12:00.000-07:00

A common need for users of Hadoop and Amazon Web Services is to move whole directories of data from an HDFS cluster (whether inside or outside EC2) into Amazon S3.

The traditional solution is distcp. Amazon has a better version S3DistCp. A common idiom is to have a script run continuously looking for complete directories that are ready to be moved and then to create "marker" files to indicate that the data push is in progress or completed. The marker files are a way of storing meta-data about the data directories. You could do this in SimpleDB or your own SQL db, or a text file or anywhere else. I like having it available right along side the data.

I thought this was a general enough pattern that I would share a bash script to do it. The code contains comments about what it's doing and why, along with instructions on installation and configuration (as constants).

Basically, it looks for directories in HDFS matching a certain pattern and moves them to S3 using Amazon's new distcp replacement, S3DistCp.

It creates marker files (_directory_.done and _directory_.processing) at the S3 destination, so that it can synchronize when multiple instances of the script are running, and so that down-stream processes will know when the data directory is ready for consumption. This wouldn't be necessary if we were moving a single file, since it wouldn't show up at the destination until it was complete. But is necessary when moving directories of files, where some files might be completely transferred, but not all.

Why I switched from iPhone to Android HTC Evo

2010-09-10T12:29:00.000-07:00

The Android HTC Evo is a great phone. I've been talking about it for a while, but I finally switched from my iPhone 4 on AT&T to the HTC Evo on Sprint.

Here are my reasons:

Evo has a larger screen (meaning bigger buttons, easier to view), but not too big that it doesn't fit comfortably in my pocket.
The Google Maps app is better on Evo. I can layer multiple search results on the same map (e.g. Show me Post Offices, Best Buys and directions to some location all at the same time). And the turn-by-turn navigation (including voice) is a nice kicker.
Mobile WiFi hot spot. I can connect my laptop or ipad over wifi through the Evo to the Internet (no tether required)
Better call quality... I frequently miss my calls on AT&T

IPhone has some nice features, but in the end these are less important to me and my life style:

Great battery life (with the Evo, I'm going to have to keep a spare charger at work and in the car)
High resolution (I'm just more interested in size, than resolution)
More attractive looking phone (guess you can't have everything)
More apps (but Android still has the basics... and who really needs 50 different ToDo list apps anyway)

Finally... don't get me wrong. I still love Apple. And I love my macbook pro and iPad.

How To Start Synergyc Automatically on Mac OS X 10.5

2008-04-04T09:44:00.000-07:00

Jan Varwig has a good post on using launchd to start the Synergy server (synergys).

If you're like me, though, you have a desktop PC with keyboard and mouse and a MacBook. So,what you really need is instructions for running the client (synergyc). With a little playing around, I came up with this variation of Jan's instructions.

Label
net.sourceforge.syngery2
OnDemand

ProgramArguments

/usr/local/synergy-1.3.1/synergyc
-f
-1
-d
WARNING
100.1.2.3

RunAtLoad

ServiceDescription
Synergy Client

And, of course, you need to modify line 11 with the correct path for synergyc. And also line 16 with your IP address (I'm not sure if you can use a hostname instead of an IP address). Save this in /Library/LaunchAgents/net.sourceforge.synergy2.plist
Owner should be root and permissions 644.

I switched from the long options (e.g. "--no-daemon") to the short options (e.g. "-f"), because I was seeing an error that synergyc did not recognize option "--no-daemon".

There is a comment from Cody Robbins about how to manually start the launchd process once you're all set up. That was helpful, but I needed the full filename for launchd to recognize it. I.e.:

launchctl load /Library/LaunchAgents/net.sourceforge.synergy2.plist

Cody's comment had the same command without ".plist".

How to move an Amazon AMI (EC2 image) from one account to another

2008-03-28T11:31:00.000-07:00

I've been having a lot of fun lately with Amazon EC2 and S3. I created a good base instance with many of the tools we needed, but it was all under my personal EC2/S3 account. In the short term, I can go ahead and submit expense reports to cover the charges, but I don't want to do that forever.

To make a long story short, I attempted to download the image and then upload it under a corporate S3 account. Luckily a few web searches and I came across this post on transferring an AMI. This saved me a lot of time... without it, I may have tried several other solutions.

In any case, I went with option 1 from the entry linked above. But they left out one crucial point. I thought it would be sufficient to start the AMI under my personal account, but bundle it using the the corporate keys. I did all that, but when I tried to start the EC2 image with my corporate credentials, it failed... basically saying that it wasn't my image. It turns out that you need to share the image with the new EC2 account. And then launch and bundle the image using that new account.

After that it worked like charm.

Google Site was JotSpot

2008-03-04T13:23:00.000-08:00

Google launched the reworked JotSpot under Google Apps a few weeks ago. I've been using it successfully, so it's a welcome addition to the toolbox.

One thing I miss though, the wiki syntax. The GUI featrues are nice, but wiki syntax is a powerful quickhand for advanced users. Wouldn't it be possible to support both? There is a button to switch to HTML mode, why not a button to enable "wiki syntax" editing features?

Domain Forwarding with Google Page Creator

2008-02-20T19:29:00.000-08:00

How do you point (or forward) your domain name directly to your GPC (Google Page Creator) web site? E.g. "yourdomain.com" --> "www.yourdomain.com". My answer follows.

I searched high and low, and had a lot of trouble finding a solution to this common usage scenario. Google's own docs, which use GoDaddy as an example, are incomplete. And a search on the web turns up a numbers of hits, but many apply to Blogger rather than GPC and others only contain bits and pieces, but not the whole solution.

Well, this is how I got it to work (hopefully I didn't break something else in the process). This may or may not apply verbatim, depending on your situation. Here is my situation:

I created a new Google Apps account for a new domain. I let Google register the domain name for me (a $10 charge). Google uses GoDaddy as the Domain registrar, automatically creating a GoDaddy account for you. If you look at the DNS configuration, Google creates something like this:

Domain Forwarding
disabled

A (Host) records
@ 64.233.179.121 TTL: 1 hour
@ 66.249.81.121 TTL: 1 hour
@ 72.14.207.121 TTL: 1 hour

These are all Google IP addresses.

CNAMES (Aliases)
www ghs.google.com TTL: 1 hour
calendar ghs.google.com TTL: 1 hour
mail ghs.google.com TTL: 1 hour
start ghs.google.com TTL: 1 hour
docs ghs.google.com TTL: 1 hour

MX (Mail) records
...

etc....

Next, I created a homepage with Google Page Creator. So, when you visit http://www.mydomain.com you got my homepage, but when you visit http://mydomain.com you got the Google Apps start page (i.e. the Google Apps portal page) instead.

Google's documentation tells you to create a forward and gives an example for GoDaddy that just has you enable forwarding with a 302 redirect and a misleading warning about the default A record. This is not enough, given the default set up as described above.

Here's what I had to do:

1. Enable forwarding with GoDaddy:

Go to the Google Apps control panel at http://www.google.com/a/yourdomain.com. From this dashboard, click "domain names", then click "Advanced DNS settings". Log in to GoDaddy using the credentials provided.

Click on your domain name, and then the link for forwarding. Enable forwarding:

Enabled
Forward To: http://www.yourdomain.com
Redirect Type: 301 (Moved Permanently)

Google's example uses a 302 "temporary" redirect, but I don't see the advantage of that over a 301 "permanent" redirect.

2. Make sure the CNAME record for www exists

If your default setup is like mine, it will already be there, but you should verify that there is a www CNAME record similar to the one above.

3. Replace the A records for Google with A records for GoDaddy

The forward enabled in step 1 will not work by default. This is because all requests are processed by Google's servers, rather than GoDaddy, so there is no opportunity for GoDaddy to respond to a request with the 301 redirect. So two steps here:

3a. Create an A record for GoDaddy

Under "Total DNS Control", click "Add New A Record" and fill in:

Host: @
Points To: 64.202.189.170
TTL: 1 hour

Presumably, if you had created your domain with your own GoDaddy account rather than letting Google do it, this or a similar record would already exist.

3b. Delete all the A records for Google.

My default set up had three A records for Google. Go in to "Total DNS Control" again, and click the [X] button (delete) next to each of the A records for Google. If you want to verify that a particular A record is for a Google server, you can copy the IP address and look it up in whois.

4. Wait for your changes to take effect (usually in a few minutes, maybe an hour).

That's all. It seems to work for me. I hope it can help you, too.

BTW, the new domain is for my wife who is a Pilates instructor in San Francisco.

New Business: Monthly Book Club For Children

2007-11-29T17:38:00.000-08:00

One of my recent projects is a children's book club. Monthly-Books.com has been live for a little while now. It's a very simple site. Sign up involves choosing a book theme, the child's age and gender. The book club will send a designer gift letter introducing the club and then one book a month for a year (assuming a 12 month membership).

As anyone who has started their own business will know, you have to master quite a few different subject areas in order to get something started. This includes marketing (in this case PPC and SEO), creative director, editorial (to write all the copy), coder (to create the pages and order processing), product (to decide on how the club should work, put together pricing models, etc.), as well as all corporate issues (book keeping, taxes, corporate filings and so on).

A major impetus for creating this site was simply to have a project where I could learn some new technologies. In this case, Groovy and Grails. I find that it is easier to learn something when you have a real world application for it.

Revision to the Book Swap Idea

2007-11-29T11:15:00.000-08:00

As previously mentioned, here's the revision to my business idea for a Book Service:

I've revised this idea based on the feedback I've gotten and research I've done. Margins on books are quite slim and it turns out that there are quite a few competing services. In fact there are so many competing services for After Market Books (and media), that it's quite fractured and confusing for the consumer. Enter PBAL, Personal Book Acquisition and Liquidation. I know... it's a terrible name. The concept is to be a facade over the myriad existing outlets (Half.com, Amazon, Ebay, Bookmooch, Bookcrossing and so on). One place for consumers to list a books to get rid of and to find the books they want.

The consumer creates a list of Books they Have, and a list of Books they Want.

For HAVE items, the service posts the items to all or some subset of Half.com, Amazon as a used book, Ebay, Bookmooch, BookCrossing, etc.. When a book is eventually sold or otherwise moved, the service would know to go out to the other outlets and cancel the item.

For WANT items, the service would search the same outlets or subset of outlets for each item. E.g. start internally by checking other consumer's HAVE lists, next look at Bookmooch (if the consumer has points available), then Bookcrossing, then check the lowest prices on Half, Amazon and Ebay Buy It Now Pricing (and maybe auctions that are near to closing).

Revenue Model
Well, since the space is so tight, I think it would need to be a Free Service. So, it would need to rely on a combination of 1) targeted ads (we know the consumer's geography and the genres of books they enjoy); 2) product sales (e.g. mailers, shipping labels, packing tape, digital scales, etc); 3) affiliate programs with Amazon and the other outlets and maybe other services like stamps.com and the like.

Final note... this model is extensible to other media (Music, movies, games, etc.).

One Week Detour in LA

2007-11-29T11:06:00.000-08:00

We haven't quite reached San Francisco, yet. We're stopping for a week to stay in Los Angeles with my brother and Tabitha. BTW, my brother's latest movie, Pretty Bird, is slated to premiere at Sundance this year.

Anyway, I don't expect to do much site seeing while here, we've visited enough times that the novelty has worn off. But we'll enjoy the weather and the company, and my wife, Jennifer, can work on completing her Pilates certification. It's good to see all our plans moving forward.

Choosing a neighborhood in San Francisco has been a chore. Top of our list are Noe Valley, Mission, Potrero Hill and Inner Sunset. We haven't seen the latter, yet, but have heard it's nice. We found a three month sublet in Potrero Hill which sounds great. I hope it's everything we expect when we arrive this Saturday. The plan is to spend a few months in each neighborhood, so we can choose what works best for us from actual experience, rather than just research alone.

Catalog Choice - About Us

2007-11-07T05:54:00.000-08:00

I found this great service and wanted to pass the word along:

Catalog Choice

They help to reduce the amount of physical junk mail that you get, by unsubscribing you from catalog mailing lists. Save time and energy (fewer catalogs being produced and shipped is one way to go green).

Time will tell if it works....

Diversifying Your Investment in Time

2007-10-31T14:13:00.000-07:00

There are many investment models set up to limit investor's exposure to risk. Commonly they are ways to diversify your investment. The Mutual Fund allows you to invest any amount of money and have it distributed across dozens on hundreds of different stocks. VCs are a way for investors to set up a fund and invest in multiple startups with the hopes that one or two will make it big. Loans are commonly packaged and resold in batches with some blended "risk" factor. REITs allow you to invest in multiple real estate projects. The list goes on. This is huge business and financiers are always looking at new ways to diversify... ways to limit risk, but maintain a great return.

Since time is money, why is there so little thought about how to diversify our investment in time. The average person takes a job. They may get some stock options. Options are inherently risky, especially for a company that's not yet public. Clearly, options have other purposes as well, but from an investment point of view it's pretty risky. What if the company goes belly up? Then you've not only lost your job, but your investment in stock options (even if you didn't exercise the option, you've made a time investment in them) is lost too.

There are, perhaps, a few strategies to deal with this now. You can do free lance work for a combination of fees and drips and drabs of equity across multiple client firms. You can liquidate your stock options as soon as they vest (assuming a liquid market) and immediately put that cash in to other stocks. You can start a side business (comes with great tax benefits): sell antique chess pieces on eBay or write restaurant reviews and earn money through Google AdSense. Serial startups: if my pet food business doesn't take off in a year, I'll cut it and try importing furniture from Indonesia.

What else could be done? Let's brainstorm for a moment...

If you're in a VC funded company, instead of taking options in your company, take options in a pool provided by the VCs which consists of startups at a similar stage. Presumably the VC has done some due-diligence for the companies they're investing in. This one is a little unrealistic as the member companies are unlikely to approve of their VC diversifying the options. They want their employees to be heavily invested in their work. This is the point of giving out the options in the first place. Theoretically, though, a third party could "buy" your options (or rather, a contract with you granting the right to the proceeds from exercising that option at some point in the future as a way around the legal restrictions on options) in exchange for options in a "fund". These scenarios are further complicated by the difficulty in valuing a non-public company.

That one's tricky. Let's explore some other ideas. If you are a founder... you could swap a small chunk of equity in your company for equity in a pool. The members of the pool have equity stakes in each other's company and therefore a vested interest. So their may be some collective bargaining power and stronger impetus to strike mutually beneficial business development and marketing deals (e.g. link exchange, affiliate programs, co-reg, cross-promotion, and other deals/partnerships). The members of the pool might also serve as an additional, in-formal board of advisors (since there is an actual equity stake, perhaps the pool members even nominate an actual board member).

Of course, this leaves all sorts of questions open. For starters, how do you choose the right pool to join with. I wouldn't want to co-mingle my fortunes and future with just anyone... the point is to limit risk, but still maintain the best possible return. One answer would be to choose 5 (for example) entrepreneurs that you've had some interactions with and start the 5 businesses in the pool together, from scratch. This way you know who you're working with and the companies are all starting at the same stage.

Clearly, each idea is fraught with a number of big questions and impediments. I don't have the answers. My goal at this point is just to start discussing the topic. Some people might say that a good entrepreneur should believe in their idea enough so that none of this matters. But, I'm convinced that diversification is always a sound strategy. And if you're not a 20 year old kid with nothing to lose (maybe you have a family to support) risk will always be one of your concerns.

Another solution that's been discussed elsewhere, is the idea of a parallel entrepreneur. Here's a good quote, from Scott Rafer from an interview for the San Jose Mercury news. I can't find a working link to the original story on mercurynews.com (perhaps they don't support permalinks):

"Basically, everyone I know is involved in five or six projects right now," said Scott Rafer, 38, former CEO of the search engine Feedster and now CEO of blog tracker MyBloglog.com, co-founder of Mashery.com, a stealth-mode company aiming to help Web developers, and chairman of WiFinder.com, a WiFi hotspot directory. "VCs spread their risk across numerous companies," said Rafer. "Why shouldn't we?"

And here are some good concerns about the parallel thing from Dharmesh Shah. And another parallel entrepreneur article with a more positive spin.

I 'd like to try this route myself, with a set of ideas that have low-start-up costs. A few of strategies to mitigate the dangers:

1. Only have one business in serious implementation phase at one time. Other phases are more forgiving for deadlines.
2. Outsource and delegate as much as possible. Decide what's really necessary to do myself.
3. While the point here is to hedge, I can always look for opportunities to leverage as well.
4. Find partners to work for equity. You're short on time, but you have "extra" equity (extra because you have two or three start-ups).
5. Much of the sage advice from 37Signals: Ignore Details Early On, It's a Problem When It's a Problem, It Just Doesn't Matter

I'll post more on the ideas and how they are coming along as I make progress.

No Longer With TheLadders.Com! CA-- here we come!

2007-10-19T00:20:00.000-07:00

I am no longer Director of Application Development at TheLadders.Com. After almost 3 good years, it's time to move on. And, of course, I was sure to exercise my stock options. I learned about Product Management, I learned technology and development process, and made lots of connections. A worthwhile way to spend those years. But now I have other plans.

For starters, Milk & Cookies is already sold. The lease on our apartment ends in November and we're selling all our stuff on craigslist and ebay. Clearly, a move is in the works. My wife, daughter and I are relocating to the West Coast. I expect to be in San Francisco by the end of November.

We're making an adventure out of it. Packing our car and taking two weeks for a road trip across country. The line up of cities in our tour: Youngstown, Chicago, Des Moines, Lexington, Boulder, Denver, Santa Fe, Sedona, Los Angeles and finally San Francisco. We'll be seeing some family along the way... Thanksgiving in Sedona works out particularly well. We haven't found an apartment in San Francisco, yet, but we don't know the town well enough to want to lock into a lease now. We'll probably find a sublet or extended stay hotel for a month when we first get out there so we can spend some time choosing the right place.

I imagine this will be a vacation like no other. Imagine a trip with no responsibilities: no mortgage, no lease, no job to worry about. Just you, your family and light possessions. A minimal plan, so you can stay and go as you please at your own schedule. I can't tell you how much we're looking forward to it.

Google Maps, by the way, has been a great tool to help plan the trip. After choosing the start and end destination, it finds a route for you. You can then add additional stops along the way, and Google will reconfigure the route to take you through those stops. There's also a great feature where you can drag a leg of the route on the map, to re-route you along a different path. I like to think of it as "thoughtful" functionality.

The Problem With This Blog

2007-10-17T10:33:00.000-07:00

I don't have a specialization, yet. In order to gain recognition or attention, it's important to specialize in something. This is good advice for someone in a new job. Become "the expert" in some domain. When people are working on X, they should think "Hey.. Tommy's an expert on the Acegi/Spring Security model... let's ask him." It doesn't have to be a broad category, but some reason for people to think of you.

I believe the same is true for a blog. There needs to be a reason for people to become regular readers... they're interested in your topic and want to follow your blog as the definitive source. Case in point: John Battelle's Searchblog. This blog is about very broad topics: Technology and Entrepreneurs. Even one of those is vastly too broad.

For now, I'll just write about whatever comes to mind, but I'll keep thinking about what my speciality can be. I'm hoping it will emerge on it's own.

Win for Cambrian House

2007-10-17T10:28:00.000-07:00

I mentioned Cambrian House here before. I got a lot of feedback on my book trading service idea. Some seemed to miss the point, which is an indicaton to me that I needed to work on the elevator pitch, but other comments were truly insightful and helpful. One of the most obvious benefits is the use of crowdsourcing to quickly identofy competitors or compatible services. In this case, I learned about bookmooch.com, which I had never run into before. It's a free service that allows swapping of books (C2C) for credits.

Based on the feedback, and my new knowledge of the Bookmooch service, I've reworked my idea in order to differentiate it. I'll post the new idea soon.

I consider this a win for the Cambrian House model.

Milk & Cookies -- SOLD!

2007-09-28T03:12:00.000-07:00

My wife, Jennifer, and I have sold Milk & Cookies Bakery. We started the bakery in 2005 as described in a prior post. Now, we've decided to do other things with our lives. For starters, Jennifer is getting her Pilates certification. I suppose after fattening the world with cookies, she thinks it's time to assist the national health average.

More about my plans coming soon...

Book Swapping Business Idea

2007-09-21T12:12:00.000-07:00

There are a couple of Netflix like services for books out there. Basically you get 3 (or 4 or 5) books at a time, as you read them and send them back more books will automatically be sent to you. Bookswim.com and booksfree.com (contrary to the name, the books are NOT free). are two examples.

The thing that makes these services difficult is that warehousing and shipping and handling books costs more than similar work with CDs. Wouldn't it be more efficient to ship the book from one consumer directly to another? Only one shipping charge, and no costs for handling or inventory storage.

I haven't decided on a specific model and I'm looking for input on what model would work best.

ASSUMPTIONS

1. The queue functionality is important. This is a good way to deal with titles that have limited availability. Also gives insight into a consumer's preferences. It serves as a reminder for the consumer of titles that they are interested in. They can add titles whenever they think of them , when it's convenient. They don't have to think about what they want "next", when they return one title.

2. Some people are collectors. I have friends that have hundreds of DVDs at home, so they can watch them over and over. They don't subscribe to Netflix. Likewise, people who want piles of books on their shelf would not be the target market for this service. Instead, we would go after the larger market of people who want to read them and get rid of them. In fact getting rid of them is a bonus, as they don't have to feel guilty about throwing them away or having to drive them over to a library (many of which has stopped accepting book donations any way).

3. For convenience, we would need to provide postage-paid mailers. Hopefully, the USPS allows prepaid postage at Media Mail rates.

4. If the consumer is responsible for shipping the stuff, how do you know they will do it? I think we would have to rely on the Honor System (http://sethgodin.typepad.com/seths_blog/2007/07/the-honor-syste.html). Most people are honest! But people who ship quickly, will get higher priority on books in their queue.

POTENTIAL MODELS
A) List the books that you have available and the queue of books that you want. When a title you have appears on someone else's want list, we send you instructions to ship it to them. We then do the same for something that is on your want list. Charge per book sent to you? Or monthly subscription charge?

B) Forget trying to get consumers to list books they have. Instead, we seed the service by purchasing the books at the top of peoples' want list and sending them. Then we know what they have (what we've sent in the past) and can have them ship it the next time that book is on someone's list.

C) Similar to B, but instead seeding the service by buying books through traditional channels, we search on half.com, bookcrossing.com and similar services. We have those people send the book to our customer. At the same time, we try to get the Half or BookCrossing member to join our service.

Cambrian House

2007-09-21T12:07:00.001-07:00

I joined Cambrian House ("Home of Crowdsourcing") today. CH is a site where you can post business ideas and get comments. Ideas are ranked through weekly competitions. It's also a community site, where they attempt to leverage the crowd to help shape and eventually build your idea.

In my visit, I posted several comments and posted one idea of my own that I'm trying to shape:
Book swap / rental

Vote:
1 2 3 4 5

The actual idea is explained in my next post, but join cambrian house and vote for my idea.

Running a bakery in Manhattan: Milk and Cookies Bakery

2007-09-15T07:48:00.000-07:00

I've mentioned before, in this blog, that I own a bakery. Here's the story.

My wife and I started this bakery back in April of 2005. The same week that I started my new job at TheLadders.com (after leaving Citigroup/Smith Barney). The idea was simple, a niche bakery in Manhattan specializing in cookies. With the perfect accompaniment: Milk. We choose organic milk from Ronnybrook Farms dairy, and gelato from Il Laboratorio Del Gelato. As another differentiator, we said why limit people to just a few varieties of cookies. Instead we can give them a "Design Your Own Dozen" (DYOD) option.

The concept works like this. Choose a base dough. The most popular is, of course, Vanilla (this is the base we use for chocolate chip). My favorite is the chocolate base (it's like a brownie in cookie form). Then choose your mix-ins (nuts, dried fruit, flavored chips, candies and so on). A great combination is the "PB&J"... peanut butter dough with PB chips and dried cranberry. Another great option is Chocolate dough with Heath and, once again, PB chips. Then we bake up a fresh dozen cookies while you wait. Of course, if the customer knows what they want, we would encourage them to call ahead so they don't have to wait at all.

DYOD turned out to be a great hook for PR purposes. Just days after launch, we landed placement in Daily Candy and the New York Times. Roker on the Road followed, along with many other TV, print, radio and web appearances. And, as a crowning achievement, "The Today Show," where we made DYOD cookies with Tim Robbins and most of the Today show cast. This was a great experience. Just so you don't think everything is rosy, though, be aware that running your own business is full of highs and lows. My wife likes to tell the story of how she was on the Today Show in the morning and then scrubbing dishes at midnight of the same day.

Any way, we've run the store through thick and thin... the Holidays are always hard. And we've managed to turn over operations to a great girl we found to come work for us. Come visit us at 19 Commerce Street in New York's Greenwich Village.

How Does Bookcrossing Effect the Small Shop Owner?

2007-09-14T07:21:00.000-07:00

I posted before about bookcrossing.com. A fun way to clear out some of your old reading material. I released another book today. A bookmark, slap a sticker on the inside cover and a post it on the front. Then off to a local coffee shop I went. It seemed a little awkward to me to walk in drop a book and run, so I bought a coffee and muffin. And it strikes me as ironic that I needed to drop $4 (I had already had coffee that morning) in order to release a free book that is probably only worth about 2 bucks in a used book store. This is even more ironic, because after-market, mass-market paperbacks are virtually worthless. Book distributors instruct retailers to destroy mass-market paperbacks and only mail back the cover when doing returns. Presumably this is because the cost of shipping is more than the cost of the book, and the cover return is just for fraud prevention.

Should I feel an obligation to the shop owner? I'm donating a book, but I don't really have any official permission to leave it there. What's in it for the mom and pop of this retail outlet? Good will? Traffic from people browsing books? I think the latter is unlikely as they only have one or two random titles, not much of a library. Many coffee shops have community bulletin boards, as a way of ingratiating themselves with the locals by giving something back. A few bookcrossings may help establish that place in the community. I'm not sure what the answer is, and as a bakery owner myself, I don't see any compelling reason to desire a few abandoned book titles. On the other hand, I don't see any reason to dissuade it, either.

One way to make this situation better would be to establish partnerships with the shops. Where they set up an "official" bookcrossing Zone. With space and signage. The zone can be targeted to a specific genre such as Mysteries or Science books or Historical fiction. Then the zone would become a niche destination. Bookcrossing.com could then publicize the zone on it's site and someone looking for a new romance to read before bedtime would know where to go. Bookcrossing gets activity and the shop gets traffic. Win-win!

Before you criticize on the obvious problem of how bookcrossing.com could possibly have the resources to form relationships with tens of thousands of venues across the world, let me propose a solution to that, as well. Bookcrossing's members would become the sales force, with downloadable marketing materials from bookcrossing.com. I'm sure the membership base contains plenty of people who like people and would be happy to approach proprietors in order to set something up in their neighborhood.

Startup's Choose Their Own Path - Founders At Work by Jessica Livingston

2007-08-29T13:35:00.000-07:00

An inspiring book. Including first person retrospectives of the history of the computer industry over the last 30 years (Wozniak of Apple Computer or Ann Winblad of Hummer Winblad from the late 70's to Mark Fletcher of Bloglines in 2004). And great stories from a wide variety of entrepreneurs with many lessons.

One point I found to be very interesting was the contradictions between individual stories. In some cases VCs are the devil (according to Philip Greenspun of ArsDigita) and in others, they're altruistic heroes (Mena Trott's story about Six Apart). In some cases, the business just fell into their lap (HOT or NOT), and in others they are the careful result of years of planning and steady progress (Research In Motion).

Pain
A common theme is that budding entrepreneurs should expect pain from their startup. "It sucks," says Joel Spolsky of Fog Creek. And James Currier from Tickle responded like this:

Livingston: What was most surprising to you...
Currier: How painful it was....
Livingston: What would you tell someone who was considering starting a startup?
Currier: That it is incredibly painful and it will take over your life.

But in another interview, Blake Ross, co-founder of Firefox, says "It's just really freeing to not be [worried]... We weren't trying to strike it rich with Firefox. It's open source and it's free. ... it was OK if we failed. ... Just product and users, all day every day." And Charles Geschke (of Adobe Systems) suggested that it's manageable: "work smart and not long".

The VC Experience
We've all heard horror stories about VCs. Many of those stories are well accounted for in this book. The most striking is Phillip Greenspun's story abut his consulting firm, ArsDigita. Greenspun says: "The senior partners in [the VC] firms were so rich they didn't want to spend time sitting on the boards of companies... The VCs delegated very junior people to sit on our board. ... [the CEO] had never run a company ... never had profit-and-loss responsibility... and he never started a company." It gets worse from here, as Greenspun unveils a story of an inept CEO and a CFO who "was really bad with numbers". Eventually the whole mess winds up in litigation.

On the other hand, some of these first hand accounts praise their VCs. Mena Trott (co-founder of Six Apart) has nothing but praise for Barak Berkowitz. Introduced by Trott's VC, Barak became a board member and then the CEO of Six Apart. "If it wasn't for Barak, I don't know where we would be now." And Arthur Van Hoff (who started Marimba) said VCs "are instrumental in your success because they give you money and a really strong endorsement. ... They are really working your case. ... it rarely happens that they turn against you, because you're a team and if the team isn't working, the company will likely fail.'

Luck
Success for some startup's was serendipitous. James Hong and his friend assembled HOT or NOT in just a few days, mainly as a gag. There was no expectation that this could be a company. But then it got 40,000 hits on it's first day and they were off and running. A few months later and they had a revenue stream (a dating service).

Another example: Joshua Schachter wrote Muxway (a predecessor to del.icio.us) as a way to tag his own collection of bookmarks and made that collection available on the web. 10,000 users, 30,000 users... and it kept growing. This was before he even decided to get VC funding and make it a company.

And then there's companies like TiVo, where Mike Ramsay (a TiVo founder), set out to make a consumer product. A home network set top box. The original idea "had a ton of apps which we thought up, one of which was DVR." And quickly narrowed down to "you can't do everything, so let's design a simple server based on very low-cost technology. Let's decide on one app that we think is the killer app." That app was the DVR that we all know as TiVo.

Original Business Plan
Many people know the story of Flickr's original business plan. To recap, Caterina Fake and her partners "were building a massive multi-player online game called Game Neverending. .... and there was a social network associated with it." Then they "just added the ability to share photographs." And "in July of 2004, [they] had to put the game on hold and stop development on it because Flickr was really taking off." So they moved from their original business plan (a game company), to a completely different service in a different space (photo sharing). This appears to be common in many start up ventures. Stephen Kaufer says of TripAdvisor, the company he co-founded, "you can't get attached to your vision in a startup, because things may change."

But others, like Mike Lazardis of Research In Motion had a vision and made it reality. Lazardis says "...I had just put the finishing touches on what became the plan for what eventually became the BlackBerry ... I came up with the basic premise as to where the value was and what became the foundational underpinnings of our technology for almost a decade after that point".

Competitors
To what extent does a new venture need to be concerned about competitors. Ann Winblad (founder of Open Systems, and currently at Hummer Winblad) says "we didn't ever worry about competition. ... nobody had any market share-- 100 percent is available for everyone, so we wouldn't get it all anyway." Nor was Yahoo's first employee, Tim Brady, concerned with competition in the early days.

But TiVo, according to co-founder Mike Ramsay, was in fierce competition with Replay from the beginning.

The Need for Cash
Common wisdom is that one of the hardest parts of starting a new venture is funding. And that's certainly true of many of the stories here. Some were close to going under before receiving another round of founding. Paul Graham of ViaWeb had this to say:

Livingston: What did you worry about most?
Graham: Running out of money

Then again, there are several others who didn't need more cash or didn't worry about it. Blake Ross from Firefox says "when we did Firefox, money was just always sort of there. There were donations, seed money from AOL ... it was a hobby, right, so we didn't really care."
And Joshua Schachter from del.icio.us goes so far as to state that too much of good thing is dangerous, "I think in general being overcapitalized is a path to failure. ... There are general ills with being overfunded."
And HOT or NOT co-founder James Hong says "...even if you raise money, spend it as if it's your own and you have none. Your organization has to remain smart and lean. Be cheap."

Who Do You Believe
There's so much good advice out there and much of it in this book. You would be a fool to ignore it, or worse yet, not seek it out and read it. But remember there are many paths you can take. The right one will be dependent on your situation and the players involved. Pick and choose what works for you.

One more comment about Founders at Work: Who else was blown away by learning that Captain Crunch worked with Steve Wozniak at Apple Computer?

Technorati

2007-08-29T06:20:00.000-07:00

I've registered this blog with Technorati. For those that don't know, Technorati has a feature where you can save specific blog posts to your favorites for later reference. And, of course, the world being what it is, you can tag the blog post with any keywords you choose to help you find it later. For example, my favorites includes this cheat sheet for web technologies and is tagged as "tech reference".

My Technorati Profile.

I'll be adding a button to the side of my blog so you can automatically add an article to your Technorati favorites.

How GoogleNotebook Can Improve

2007-08-17T15:08:00.000-07:00

I've pretty much abandoned Outlook for Personal Information Management in favor of Google's various apps. Calendar and Email are no brainers. But the GoogleNotebook deserves some further comment. When you consider the typical functionality of a notebook, it should be the easiest app to provide, and in fact there are many competing implementations from other providers. There's much more that should be done, though, and it seems to me that because the basic app is so simple it get's very little attention.

The existing product has some good functionality: multiple notebooks and sections to help organize your notes, nice AJAX UI to move and browse through notes and edit with basic WYSIWYG controls, sharing so you can collaborate or make your notes public. The trash can, too, is a nice addition. And they've made progress since the early releases. At first, it was very URL centric. As if the only thing you could possibly want to make a note about is a web page identified by a URL.

As a side note, as I come up with blog ideas, i enter them into one of my notebooks, and in this case, I'm even writing this entry with GoogleNotebook.

Accessing the Notebook
I chose to abandon the Firefox extension, as, again, it is focused on making notes about web pages, which I don't often have a need for-- usually a simple book mark is enough. Instead I added the notebook widget to my iGoogle homepage. The module is a little buggy. I frequently can't insert the cursor on a note and type. I can use it to browse notes, but when i want to enter something, i need to click on the header which will take me to the Google Notebook site.

UX
It can be tricky to jump to the bottom (or top) of a note to enter something at the end. I usually just hold the down arrow and get to the bottom of a doc, but in Google Notebook, when you are at the end of a note and click down, it goes on to the next note. [I just realized that PageUp and PageDown work and will not take me beyond the note I am currently in.]

I haven't found a reason to use "Add Comment" at the end of a note. Why not just enter your comment inside the note? Anyone have a use for this feature?

If you think about how a physical notebook is used, some common use cases include images. E.g. a scrapbook or notes with sketches. I should be able to embed (inline preferably) images and other document types in my notes. This would include JPGs, Google Docs, Spreadsheets, PDFs, etc. And a drawing interface to include simple sketches would be a useful feature, as well.

What about tagging? Google is a big supoprter of the tagging model (a la Gmail), but it's no where to be seen in GoogleNotebook. Sure, I can organize my notes in multiple notebooks and sections, but that's a hierarchical organization and at this point we're all aware of the organizational benefits of the tagging alternative. I want both options.

Integration with other Google Functions
I'm also disappointed with the level of integration between the Notebook and other Google functions. For example, I don't make notes about random URLs, but I might want a note about a GoogleBookmark. Note that this is different from a Note with bookmark in it.

Gmail presents another good opportunity for integration. I often get emails with bits of information that I want to save. Frequently I just want to file away a small snippet from the email rather the whole email. Why can't I make a Note from an email message or a portion of an email message.

Conclusion
In the end though, I'm using the product and realizing benefit from it. I have some complaints and my wish list, of course. I'm just not blown away by it, in the way that I am by Gmail or Google Calendar. To be fair, though, I can see improvements rolling in over time.

Is Netflix in a commodity business?

2007-07-26T13:17:00.000-07:00

I got an email from Netflix yesterday that they were reducing my monthly rate: "We're lowering the price of your Netflix plan!". Great news, but I have to wonder why they are doing it? I already had a good deal, because I signed up nine or ten years ago, I have the original plan (4 DVDs at a time) for the price of the 3 DVD plan now. I love the service and I've never had an issue with the price. I was perfectly happy paying $20.99 a month. Or maybe it was $23.99? I don't really know, nor did I really care. The service is great, what matter a few bucks. The convenience and the quality of the service has always been much more important than the price. And in my mind, it's a substitute for paying for premium cable channels. We only have basic cable.

My favorite Netflix feature is the queue (which was around from the beginning). It's great because I can add movie titles whenever I see an interesting movie poster on the side of a New York City bus stop or hear about a cool old movie from a friend over a cup of coffee. I don't have to think back about what I was interested in seeing in order to pick my next movie. I can even add movies that aren't yet available on DVD. A brilliant move... I want to add titles as I think of them and not have to keep another list of things I want to add ("my queue of things to add to the queue") at some time in the future. I probably have 40 or so items in my queue now, many of which are new or recent releases in the theaters.

Netflix has great customer service, too. For example if a movie gets lost, they replace it immediately... no questions asked! A great example of The honor system at work.

So I had to wonder: why drop the price? I think a more typical marketing approach with regard to price cuts is to change the price for new customers, but leave existing customers where they are. Such a change would help acquisition, but not retention. So changing the price for existing customers, must be more about holding off defection to copy cat services. I.e. Block Buster, etc. I haven't tried these other services, but I have to wonder if the industry is really at the point of a commodity, where the key differentiator is price?

The recent Q207 Quarterly Earnings Call for Netflix sheds some light:

Our strategy evolution is shifting some profits into growth investments for as long as our competitor runs their online service at a considerable loss. The tactics of our increased growth investments are higher service levels, lower pricing and slightly less marketing because we believe that is the most efficient combination for Netflix at this point.

I hope this strategy works out. I have an emotional attachment, since they were the first in the market. I'm sure the industry will evolve to a download or streaming model. And Netflix will still be at the forefront of that movement. The delivery method isn't critical to this business model. Instead, the important factors are the queue, the recommendations, customer service, ease of use and so on. This evolution will also make Block Buster's one advantage (the ability to swap a DVD at a local store) obsolete.

What To Do With Old Books

2007-07-08T04:46:00.000-07:00

I happened upon this totally unexpected, very creative service at my local coffee shop the other day. Sitting on the coffee table was a novel with this note on the front.

Bookcrossing? Let me flip through. I quickly found a bookmark and a bookcrossing sticker on the inside front cover:

What a great idea! I have lots of old books, but I prefer a lighter lifestyle. Less baggage, it's good to be mobile. This is a new found desire, so coincidentally; I just dropped off a couple of boxes of books to the local library as a donation. I could take the rest of them there, but I like the idea of the low overhead distribution of bookcrossing. Plus, with this method I can follow the path of each book as it makes its way into the hands of new owners. Evidently, they've been around for a while with half a million books in 120 countries, but I've never run into them.

Let the journey begin. After signing up with bookcrossing, the first step is to "Register" your book. This is where each book gets its own book crossing id (or BCID) and is added to the bookcrossing database. You enter author, title and category info (there is a box to enter ISBN and have the book info looked up on Amazon.com, but this didn't work for me). You can also rate the book and add comments or a review if you like.

Now you need to label the book with the BCID. You can print labels and stickers, or buy them from the bookcrossing supply store. I did the latter, as I want to support the concept. I bought a 25 pack (25 each of a bookmark, a sticker for the inside front cover, and a sticky note for the outside). They arrived 3 or 4 days later.

Next step is to "Release" the book. This simply means to drop it in some public space where it can be found by an unsuspecting stranger. At the web site you mark the book as released and make a journal entry of where you released it. If you want, you can receive email whenever someone else catches your book or makes a journal entry about it.

I've released my first book at another coffee shop. It's a fantasy novel I read years ago. I'll release the others at different locations around New York City, and maybe some in other cities I visit.

The one oddity is the "Tell-a-Friend" service. There are links for common mail clients (online and offline) to help automatically load friend's email addresses, such as Gmail and Outlook. If you click Gmail, for example, it requests a username a password with assurances that they don't save this information. This is probably true, but it's a terrible practice. Web sites should not be encouraging the casual disclosure of personal data. Thankfully, you can bypass the email clients and just type in email addresses directly.

Me: Technologist and Entrepreneur

2007-07-05T11:14:00.000-07:00

This is a place to record my thoughts on all things. In particular, I spend most of my time thinking about technology and entrepreneurial topics, which nowadays are almost one and the same. As a personal blog, however, I do reserve the right to post on any topic, including my personal life.

The "X" in the blog title refers to the cross-product of the two domains in the title. There are a number of Meetups, discussion groups, web sites and other community forums devoted to "entrepreneurs". It seems to me, in this Web 2.0 (second dot-com boom) world, that most of these cater to and focus on technology (i.e. businesses based on using new technology). This tends to be the case for me, as well, but I'm making it explicit here, as a reminder that there are entrepreneurial endeavors of another sort.

For example, my wife and I started Milk & Cookies Bakery in New York City. It's an entrepreneurial effort. A successful one, featured on the Today Show and many other press outlets. We use technology to run our POS system, to advertise (Google Ad Words, MSN AdCenter, etc), to run our web site and track traffic (Google Analytics) and more. But it's not a technology business... it's a cookie business. We bake and retail cookies, brownies, ice cream, milk shakes and so on. If you're in Greenwich Village, check it out.

On the other hand, while I've started an old-fashioned retail bakery, my entrepreneurial interests still lean towards technology and software. I choose to work with Start Ups. That is, growing businesses with lots of challenges and frequent change. I'm currently Director of Application Development at TheLadders.com. A rapidly growing start up that is redefining the business model for Job Boards. Serve the Job Seeker instead of the employer. We have a responsibility to the Job Seeker, who pays for superior job board service.

Many of the posts that follow will talk about ideas that have been employed at TheLadders.com. There is a developer blog at TheLadders.com, with several of my posts. I may cross-post these articles here.