Flaptor

We just launched a new version of Twist with a number of new features:

- Updated look and feel. The page now displays percentage values (% of tweets that mention the concept) and highlights the intervals during which the concept was hot.

- Zooming in to a portion of the chart (by clicking and dragging the cursor) and getting information for a specific time (single-clicking a point in the chart).

- Context: we added a pane that shows tweets about the concept at a given time (peak or not). Also, when possible we display a brief explanation/definition of the topic (retrieved from Wikipedia) along with a link to the article.

- Browsing all hot topics from the past seven days. As hot topic detection is probably the most interesting and useful feature of Twist, we put all the hot topics in one page. There is a daily/weekly view and much more detailed information for each topic. Also, you can see what was hot at a given moment. When clicking on a point in the chart, the new pane shows other topics that were hot at that time.

- Better embed. It is now possible to embed a dynamic or static chart, choosing between two different sizes.

There is more, take a look at Twist’s Help and New Features.

A post on Techcrunch questions Marissa Mayer’s assertion that “search is 90 percent solved”. There is an interesting discussion in the comments about where search is going, but there is something more fundamental that is not addressed. Let’s leave the 90% progress mark aside for the moment. What does it mean for search to be solved?

In order to solve a problem, we must first know what the problem is. How is “search” a definition of a problem? For many technologies it’s relatively easy to find potential improvements that could be considered problems to solve. As an example, let’s look at cellphones.

It seems obvious that there are many limitations of cellphones that stand in the way of useful applications. Battery life is an obvious one: an ideal cellphone may have a battery that lasts as long as the device itself, pretty much like a wrist watch. Perhaps it could be powered wirelessly. The point is that the user would not have to worry about limiting usage until the next recharge. Also the ideal cellphone would be extremely light, it would have a faster user interface and a better input mechanism, etc. It’s even easier to analyze the limitations of cars: lower prices, self-driving capabilities, better safety features are some issues that come to mind.

Now let’s think about Google’s web search. If we look at the problem of indexing all the text on the web so that a reasonable person can find a given document, how much better could it be? It’s already free for the end user, so “cheaper” is out of the question. It’s also hard to imagine a faster search, as Google returns results within a fraction of a second. In terms of coverage, there is a possibility that Google won’t find a document that is out there but its crawlers have not discovered yet. They have gone a long way in this respect and today it’s hard to make the case for the usefulness of a document that Google has not discovered within 24 hours of its publication. As for relevance, sometimes it can be hard to find a document that Google has indexed even after a reasonable number of tries with different combinations of keywords. I know this has happened to me in the past few months but I can’t remember a specific example (I think it was some product with an extremely generic name). Still, in my experience it’s very rarely an issue. In conclusion, for a certain definition of the problem it’s reasonable to state that it’s 90% solved.

Of course, it’s possible to describe a whole class of problems (many still unsolved) that can be labeled “search”. We mentioned some of them a while ago. Search for extraterrestrial life is an extreme example.

The main point of this post is one that happens all the time in software: someone creates a solution to a perceived problem. This solution uncovers other needs or problems. There comes a point when people lose track of the problems and it becomes very hard to discuss solutions. We can’t argue whether search is solved or not. Let’s understand and describe each specific problem first.

Thanks to al3x and ev from Twitter, Twist once again is receiving a significant volume of tweets. This means that in a few days it will be possible to see more detailed weekly trends and hot concepts. It’s already better for intra-day trends, check out the resolution for the past 24 hours:

We have been keeping track of the public Twitter timeline for a few months now. Until a few days all Twist showed was a 7-day window, but now you can select monthly and historical trends. For example, it seems that in late April people start talking more about the summer than about the spring:

Also, you can see weekly patterns. Check out how weekend is mentioned more around Labor Day Weekend.

Stay tuned for more updates. As always, any feedback is welcome!

Web apps are a lot like TV shows. In particular those that are supported by ads, because their source of revenue is the same and the viewer/user is not really the customer but the product. These days it seems that everyone has some advice for Twitter about possible business models. I really don’t think the folks at Twitter need to worry about that, if they don’t find one that works they should just focus on making the company a good acquisition target.

If you come up with an idea for a TV show and even put together a pilot, nobody asks what the business model should be because it’s obvious. A TV show is made to be sold to a network, who will in turn monetize it with ads. Why can’t you just skip that step and monetize it yourself? Simply because you don’t own the infrastructure required to broadcast a TV show. You could try to build it, but clearly it doesn’t make sense if all you have is just one show.

Now look at Google, Yahoo and MSN as the multimedia networks that they are. They have a system in place to monetize many of their offerings. Plus, they benefit from the economies of scale. Today you wouldn’t start a company to build something like Yahoo Finance. I don’t know how much revenue Yahoo Finance generates, but it doesn’t need to be much to be profitable given that the incremental cost for Yahoo to keep it running isn’t very high. Gmail is probably not very profitable but it’s still very useful to Google (brand awareness, data mining, playground for new ideas, etc).

What could a company like Google or Yahoo do with Twitter? In a way Twitter has a “search” component to it. You can ask people a question such as “what do you think about X?” and if you have a few followers you’ll probably get some answers that may start a conversation to refine the “results”. You cannot do that with an automated search engine. I’m sure it’s possible to use your recent tweets as one more input to give you better search results and show you more relevant ads (not necessarily on Twitter but on other applications). Also, there is a lot of information worth mining in Twitter but you don’t need to acquire the company to extract it :)

In short, building a product to sell to one company is a valid business model. It’s risky, like creating a piece of art that perhaps nobody will like. You only have one product that you can sell to at most one client. As long as you understand the risk and have investors who believe in your idea, there’s nothing wrong with it. Perhaps it’s a good time for large companies to start thinking of new applications like Television pilots. They should encourage small teams with good ideas to work on “pilot” web applications (like what Ycombinator does, but perhaps with a larger budget). If they like the “pilot”, they already own a chunk of it and can acquire the rest.

When I think of a successful web application, I picture one that has millions of users and runs smoothly (it works fast and it does not crash). Of course, very few are like this. Perhaps the first example that comes to mind is something like Google Search. In the following chart, Google Search is in the deep south and far east:

The above chart depicts four possible stages of the life of a web app. My guess is that the vast majority of all web apps never leave the northwest. Someone throws some php (or Ruby, whatever) together with a database schema and has a proof-of-concept up in a few hours or days. In many cases this is just fine. Maybe your boss wants some internal tool to keep track of inventory, or perhaps you have an idea that you want to show potential investors to see if it’s worth pursuing.

Some applications are in the northeast. This is usually a bad place to be, because it means that you spent a lot of effort creating a highly scalable application. Perhaps you and your team had long architectural meetings in which you imagined how you would take over the world. Your app would be distributable, you came up with an efficient database design and perhaps a caching mechanism for when you get dugg or slashdotted. As it turned out, you built it and they didn’t come. The uptime command on your server shows a system load of 0.00, and you could invite all your site’s visitors to have dinner at your place.

In the southwest quadrant we have applications like Twitter today, or Friendster a few years back. Because these applications need to grow more or less proportionally to the square of the number of users, things can get out of hand pretty quickly. Regardless, this is a good problem to have, as moving from the southwest from the southeast tends to be easier than moving from the north to the south. You’ve built something that people like, so you may be able to convince others to give you the funds you need to organize the move.

Of course the chart is a continuum. Twitter is still far from the southernmost point, and perhaps it will never get there unless the move to the east goes well. Where is your web app on this chart, and how do you plan to move to the southeast?

Twist got a lot of traffic today, as it made it to the top of del.icio.us/popular. Here are very positive comments from del.icio.us users. It’s fun to look at the logs and see what people are comparing. Also, moving it to Amazon EC2 was a good idea as the traffic is not a problem at all.

We just released a fun tool called Twist. If you are familiar with Google Trends, it should be obvious what it does. It allows you to track trends on Twitter over the past week, with a granularity of a couple of hours. While testing the tool we saw spikes in the charts whenever an interesting story broke (especially about technology or celebrities). We discovered trends such as the fact that people who Twitter tend to have lunch rather than breakfast on weekdays, but both are mentioned equally on weekends:

http://twist.flaptor.com/freq?gram=lunch,%20breakfast

This tool is coupled with our Twitter search engine, so you can see what people are saying about the concepts shown in the charts. We really like Twitter and we are excited about this project, so stay tuned for more improvements.

By the way, both applications run on Amazon EC2.

Wordpress just launched a blog search service for the almost 3M blogs they host on the site. It’s powered by Hounder, the open-source search engine we have been developing at Flaptor over the past few years. We’ve been working on this for a few months so it’s great to see it live! It’s been a pleasure working with Matt and Toni to make this happen, and it will be fun to see how many blogs get indexed in the future and how Hounder scales to handle the traffic.

We just released a WordPress plugin for our automatic tagger. It’s called TagMahal and it’s available here: tagger.flaptor.com/tagmahal. If you have a blog powered by WordPress, it will suggest tags for a post as you write it. Try it, it’s fun!