We are experiencing performance problem across all Zoho services right now (as of 6:30 AM Pacific Time). Our team is looking at it.
Please use read-only versions, which are hosted from our separate data center.
These are the URLs:
Zoho CRM: https://crm-ro.zoho.com
Zoho Creator: https://creator-ro.zoho.com
Zoho Invoice: https://invoice-ro.zoho.com
Zoho Mail: https://mail-ro.zoho.com
We are working on this right now, and we will post updates as soon as we have additional information.
Update: We are experiencing packet loss on our network access paths, and we are working on it right now.
Update 2 9:13 AM PST: We have brought down the degree of packet loss, but still due to intermittent packet loss services are slow. We are working on it.
Update 3: As of 9:30 AM PST packet loss has been reduced to negligible levels. Services should be accessible now, but we are still monitoring closely.…
Zoho’s strategy from the beginning has been to provide a wide portfolio of online applications so that you can focus on your business while we take care of the rest. Over time, we have launched many applications that support this mission. But from time to time we need look back and assess how those products are doing in relation to our expectations. Even the most well-tended gardens need some pruning every now and then!
We have decided to sunset some Zoho services so that we can focus our energy and delight our customers with our other products. Today we are announcing we will close
- Zoho Challenge
- Zoho CommentBox
- Zoho Planner
- Zoho Plugin for Office
- Zoho Share
- Zoho Viewer
Customers are, as always, our #1 priority. So you have plenty of time to get your data out we will continue to allow access to these products (in read-only mode) until December 31st, 2012.…
In 2008, I wrote a post Why We Compete with Google, in response to the persistent question at that time “How can you guys survive Google in the online office space?”
My basic argument was that business software, due to its extensive sales and support requirements, simply does not have the productivity or profitability of consumer internet businesses, and could never produce the margins that Google enjoys in search or what Facebook enjoys in social networking today or for that matter how today’s margins at over-the-hill Yahoo and getting-there eBay compare to the still-hot Salesforce. I concluded that post (keep in mind it was written in 2008) with:
When push comes to shove – and there is a lot of very messy push and shove in the business software market - Google’s resources are going to flow into figuring out how to monetize the humongous traffic of YouTube or compete in online auctions, rather than figure out a way to squeeze a bit more margin compared to Oracle or Adobe or Salesforce.
One of the priorities we set ourselves after our outage is to get a read-only instance up quickly in our secondary data center in the New York metro region. We have always had the data backed up in that data center and our software team has been busy making it run in read-only mode. Today, we are happy to announce that Zoho CRM is running in read-only mode from our secondary data center.
The URL is https://crm-ro.zoho.com
We will keep this read-only mode accessible to customers all the time, so that in the event of an outage in our primary data center, your access to data will be preserved. We still have some limitations we are working through – for example, database information is synchronized immediately between our silicon valley and New York data centers, while document attachments have a one hour lag, which we will be addressing soon.…
On Friday, January 20th, we experienced a widespread outage that affected all Zoho services. The outage started around 8:13 am Pacific Time. Zoho services started coming back online for customer use at 3:49 pm, and all services were fully restored at 6:22 pm PST. We absolutely realize how important our services are for businesses and users who rely on us; we let you down on Friday. Please accept our humblest apologies.
The cause of the outage was an abrupt power failure in our state-of-the-art collocated data center facility (owned and operated by Equinix) in the Silicon Valley area, California. Equinix provides us physically secure space, highly redundant power and cooling. We get our internet connectivity from separate service providers. We own, maintain and operate the servers and the network equipment and the software. The problem was not just that the power failure happened, the problem was that it happened abruptly, with no warning whatsoever, and all our equipment went down all at once.…
We had a 3.5 hour outage of Zoho Mail, Zoho Support & Zoho Books, between 8:45 AM and 12:15 pm PST on December 14. First of all, I want to apologize to our customers and our users. We let you down and we are sorry. We know how important it is to have access to the vital business information you entrust with Zoho; our entire company runs on Zoho applications, so we understand this in an even more intimate way. With a view to preventing such incidents and improving our response to them when they do happen, we reviewed the root cause of this outage and our response to it. This post provides a summary.
The outage arose from a simple configuration gap between our software applications and the network. One part of the health-check mechanism built into our software (ironically, the very part that is designed to prevent an outage impacting customers) made an unneeded reverse DNS request to resolve an IP address to a name.…
We have restored all Zoho services. They should be working normally. Our teams are monitoring the situation closely, and if you encounter any trouble, please let us know.
We know this is not what our customers and users expect from Zoho. We let you down today. Please accept our humble apologies.
We have launched a full investigation of the root cause, how we responded to it, what we could have done better to both avoid the problem and how we could have resolved it sooner. We will post this report as soon as it is ready.
Update: Our preliminary information is that reverse DNS look up failed in one of our subnets, which caused the outage on some of the services. We still are trying to determine why it failed, why it didn’t trigger other type of alerts and why this failure resulted in such a service outage. Once the incident is fully understood, I will post a detailed report.