Services Are Up, Root Cause to Follow

Posted by Posted on by

We have restored all Zoho services. They should be working normally. Our teams are monitoring the situation closely, and if you encounter any trouble, please let us know.

We know this is not what our customers and users expect from Zoho. We let you down today. Please accept our humble apologies. We have launched a full investigation of the root cause, how we responded to it, what we could have done better to both avoid the problem and how we could have resolved it sooner. We will post this report as soon as it is ready.

Update: Our preliminary information is that reverse DNS look up failed in one of our subnets, which caused the outage on some of the services. We still are trying to determine why it failed, why it didn’t trigger other type of alerts and why this failure resulted in such a service outage. Once the incident is fully understood, I will post a detailed report.¬†


Some Zoho Services Down, Please Check our Twitter Feed @Zoho for Updates

Posted by Posted on by

As of 9:05 AM Pacific time, we started encountering difficulties in many Zoho services. The services affected are Zoho Mail, Books and Support, as well as some sporadic issues in other services. We have narrowed it down to network issues in our data center, and we are analyzing it. We will restore services as expeditiously as possible. 

Please check our twitter feed:!/zoho  for updates.

We apologize for the inconvenience.
Update: as of 12 noon PST, we have restored Zoho Mail, Support and Books. We still are not completely out of the woods, and we are monitoring all services closely. Meanwhile, we are also looking at the root cause of today’s outage. We will make a detailed post as soon as we determine the root cause, and we will outline the actions we are taking to make sure this does  not recur. Please accept our humble apologies.

Why I Am a Businessman and Why My Employees Are My Heroes

Posted by Posted on by

(picture courtesy: The Hindu online)
That scene above is from the suburb of Chennai (Tambaram) where I grew up, where my parents still live, but in reality, it could be anywhere in India. We get monsoons this time of the year every year, yet, every year this is how it looks for a few weeks. 
I am in business because I believe there is really only one solution. We need 25x more businesses, 25x more jobs, 25x more infrastructure before India could be considered a livable country. I hope to live to see a day when one city, just one fucking city, in India will offer a world-class quality of life. Today, almost no city in India comes close to offering what would be considered an acceptable quality of life. That is why I wake up everyday and go to work, because my dream is to create sufficient profits to directly fund the infrastructure we need to live a decent life.
That crazy idea, directly funding infrastructure out of profit, would practically get me thrown out of my job in a nano-second if Zoho were a public company. That is why I don’t take venture capital and won’t ever take my company public. The good news is we have decent profit, and as we grow, my plan is increasingly becoming less and less crazy. 
But it is not really about me. I live the good life. It is my employees in Chennai that are the true heroes. Every single one of them go to work under these conditions – and I am going to be there tomorrow. Scenes like these are literally everywhere in Chennai. Our people write code,  support customers, teach and learn from each other, all under these conditions. The fact that our people ship the products they do under these conditions is a testimony to the resilience of the human spirit. I grew up in exactly these circumstances, but I still think of it as nothing less than a miracle that we are able to do the work we do.
My employees are just like people in that picture. The are hard working good people, forced to live under a broken system. Governments in India are famous for a singular lack of vision and imagination. In that picture, for example, you would think the local municipal body would be responsible for the civic infrastructure. No, that would be too obvious and too sensible, and of course that is how any functioning system anywhere in the world would work. In India, financial responsibility for that road would be divided between the state government, which manages the affairs of only 70 million people and the central government (because there is a railway line in the picture, and railways are all run by the central government, including the local train network of Chennai). The local municipal entity is powerless and broke, and it exists to basically receive petitions and forward them to the state or central governments – well, when they get out of their wheeling and dealing to getting around to doing any work at all.
We have ministries and departments for everything under the sun, from condoms to condiments, from rain forests to railways, from fisheries to fertilizers, from information technology to imaging satellites. Except that we don’t focus on basics like roads, sanitation or drinking water, because, well, that would be too obvious and too sensible. Ronald Reagan’s dictum “Government is not the solution to our problems, government is the problem” applies with astonishing force and clarity in India. That is why I am a businessman.
PS: You want to see more pictures like this, here is a slideshow from the Hindu.

How a Cancelled Project at StorageTek Led to Zoho CRM

Posted by Posted on by

I have never shared this story in public before, but I wanted to do this after reading Hank Williams’ post (via Hacker News) on race in Silicon Valley. My post is not about race, but it is somewhat related, as you will see below.

First a bit of history. In 2003, our company’s only business was network management software. Our main product at that time was Web NMS, which continues to be a successful and profitable division for us to this day, but at that time, it was the only product line we had. Zoho was not yet born, in fact, we had not even conceived of the name yet. Web NMS had many large equipment vendors as customers, and among them was StorageTek, a multi-billion dollar back-up & recovery products company, which was later acquired by Sun. After a comprehensive evaluation over several months, StorageTek chose our Web NMS product to build remote monitoring capability into their storage equipment, and they also wanted us to set up a team to customize the Web NMS so it could talk to their equipment and provide the specialized reports needed. This project began in 2003, and it was going smoothly. We had a team of about 10 engineers working on this project. We had reached a stage of trial deployment in a few customer sites.

That was when they had a big management change at StorageTek. The Vice President in charge of this project was replaced with a new person. One of the first things the new VP noted was that Web NMS came from an Indian company. Just on that basis, he instructed his purchasing people to terminate the contract immediately and award the contract to another company. The purchasing person, who shared the reason behind the termination with us, felt it was a massive waste of resources, but he was overruled. In fact, because the termination was for no fault on our side, they paid for all the deliverables, even though they were not going to use any of them.

It was sometime in August 2003 that we were notified of the contract termination. It was abrupt – as I remember it, it was on a Thursday we were notified, and they told us we should just stop the project right-away and yes, we could throw away the code. Our team, which was working hard on this project, was devastated. I assembled our team and told them to take the Friday off, and come back fresh Monday. I promised there would be an interesting new challenge awaiting them Monday.

During that year, two threads were running in my mind: how to diversify the company from its dependence on a few large customers like StorageTek, where sales cycles were very long, and decision making was often very political, something we got a perfect demonstration of. Second, I was also thinking about the emerging software-as-a-service market. We were customers of Salesforce, and while I liked their product delivery model, I felt the product itself was massively overpriced. As I analyzed Salesforce, I felt their high price was due to their business model bloat i.e overspending massively on sales and marketing. As a software engineer, I felt we could build a better product and as an entrepreneur, I felt we could cut the bloat in the business model and offer the product at a more affordable price. From my perspective, all this had the added merit that we would target small and mid-sized customers first, which would let us avoid the long sales cycles and the politics. That was just a thought, an idea, without yet a plan of action.

When StorageTek terminated our contract, that thought just crystallized. On Monday morning, I called a meeting and told our team “You are now our on-demand CRM team.” The engineers were incredulous. One of the lead engineers let out “Sridhar, we know nothing about software-as-a-service or about CRM” – I said “Well, you will know soon!”

Today, Zoho CRM is the fastest growing part of the Zoho suite, it is nicely profitable, and it is starting to give Salesforce a run for their money. Zoho CRM also paved the way to our emergence as one of the most comprehensive suite of business applications in the cloud. I guess we should thank StorageTek.

Back to StorageTek – the company they selected to replace us collected a few million dollars and never really delivered on that project. The ultimate irony was that they called us a couple of years later to help them with this project again; needless to add, we declined.

The Product Business is Like the Movie Business

Posted by Posted on by

I read the cover story in Forbes on the success of Dropbox, which is set to do about $240 million in sales in 2011, with only 70 employees. As Forbes points out, that is about 3x the revenue per employee of Google, which is no slouch in the revenue per employee department itself. First, congratulations, Dropbox! This is the type of breathtaking number that makes the ordinarily successful companies like, well, Zoho, to wonder “What are we doing wrong?”

In our 15 year history in Zoho Corporation – which is bigger than the Zoho product suite itself – we have shipped over 70 products, of which we would say about 30 have been successful in the sense of being nicely profitable. Yet, even with that group of 30 products, we have seen the 10x effect: a set of two products that have taken approximately the same amount of effort to build, by similarly situated teams, yet one of them does 10x the sales of the other, with both of them being profitable. Of course the 10-bagger is much more profitable but the key point is that both of them could be counted as successful in the sense of being profitable. We have even seen 100x difference for approximately the same effort, but in our case, that is the difference between doing only $100K a year in sales vs $10 million a year, and I would not count that as 100x because the $100K product either grows up or we would eventually discontinue it because it is not profitable.

Dropbox is a logical extension of this phenomenon, where a product does 100x the sales, without taking much more by way of engineering effort than a profitable 1x product. And then the grand daddy of them all – Google search, which in its heyday reached $1 billion in sales, on not much more than the effort of a single engineering team – the headcount gets added later to diversify the company but the original search was a small team. I believe there has only been one Google search so far, so the ordinarily successful (ahem!) shouldn’t feel too bad.

Y Combinator, which has funded over 300 companies so far, is a perfect illustration. All these teams are similarly situated, with similar founder profiles and they all get similar initial funding, and they spend similar initial effort. If we consider only the universe of profitable YC companies, my guess is that so far there is only one 100-bagger i.e Dropbox, in the YC portfolio. Based on Zoho experience, I would estimate YC has about ten 10-baggers, and about fifty one-baggers (i.e just about profitable).

Welcome to the product business, which looks very much like the movie business!

If you have a smart enemy and a dumb friend …

Posted by Posted on by

… dump the friend and embrace the enemy. 

Several years ago, we had many rounds of meetings with Yahoo on ways to cooperate. It was very clear that Yahoo had lousy leadership. On the other hand, I was quite impressed by the Google leadership I met. 

What did we end up doing at Zoho? We never pursued a Yahoo partnership, and we made our peace with Google. Yahoo just demonstrated, yet again, that we made a wise choice. Dumb friends are more dangerous than smart enemies.

Apple’s Revenue History

Posted by Posted on by

Apple’s revenue now is at a run rate of over $100 billion dollars, a good deal higher than that of Microsoft. Apple’s profits also well exceed Microsoft’s profits. Yet, Apple has only about 50% of the headcount of Microsoft. These things were utterly unthinkable just 5 years ago. One way to visualize this is to imagine Yahoo overtaking Google in the search market and in terms of overall revenue & profits (Google does about 6 times the revenue of Yahoo), and multiply that difficulty by 10-fold, because Yahoo actually still makes money, but Apple was losing a lot of money when Steve Jobs came back in 1996.

I assumed that Apple grew steadily from when Jobs came back. I was wrong. Here is the revenue trajectory of Apple all the way going back to 1981. Focus on the years after 1997.

Notice the crisis in 96-97-98 when he came back – that was a stomach churning drop in revenue, which explains why he came back. But notice that 5 years after he came back, in 2002, Apple still had less revenue than in 1997 (in fact, it was less than they had as far back as1989!). During those 5 years between 1997-2002, revenue first went up, and then went down again during the recession in 2001-2. Even worse, Apple was again losing money in 2001, a full 4 years after he came back. Why is that relevant? Any normal person would have found it very hard to keep the spirit and motivation up at that time. Apple’s greatest years were ahead of them, but only one person would have possibly believed it – Steve Jobs. No one else, in 2002, would have believed Apple’s best days were ahead. From only 2005, did Apple really achieve that big burst of growth, and within just 6 years Apple has overtaken Microsoft.

Also notice the extreme variation in growth. Even after the iPod arrived, Apple’s growth rates has dipped as low as 14% and has gone as high as 68% (until 2011, when it scaled new highs). In the 2009 bust, Apple dipped to only 14% year-over-year growth. Yet, in 2011, driven by the boom in iPad, Apple is exceeding 80% growth without meaningfully adding any headcount. This is the track record Jobs has piled up. I don’t know it will ever be equalled in this industry again.