deirdre | Ellora’s Cave: IT Infrastructure Statement (Reply)

Recently, on the #notchilled hashtag, we heard about Ellora’s Cave’s IT system in the days of yore. Then Ellora’s Cave sent an internal email on the subject. I have some commentary after that.

First, the tweets that started it all:

http://twitter.com/JetGibbs/status/566743998749212672

http://twitter.com/JetGibbs/status/566744089253920768

http://twitter.com/JetGibbs/status/566744250436820992

http://twitter.com/JetGibbs/status/566744431534288896

Note: this happened in the 2003-2004 time period. Okay, now that we’ve got that cleared up, let’s read the rest of BJ’s tweets on this topic…then Ellora’s Cave has an official response.

http://twitter.com/JetGibbs/status/566745792367841280

http://twitter.com/JetGibbs/status/566749941285289984

http://twitter.com/JetGibbs/status/566759414959718402

http://twitter.com/JetGibbs/status/566759743973498881

http://twitter.com/tejasjulia/status/566759740395384832

http://twitter.com/JetGibbs/status/566760161638113281

http://twitter.com/JetGibbs/status/566798217653026816

http://twitter.com/JetGibbs/status/566798339325587456

http://twitter.com/JetGibbs/status/566798517445079041

http://twitter.com/JetGibbs/status/566798659208347648

http://twitter.com/JetGibbs/status/566798821972541440

http://twitter.com/JetGibbs/status/566798928071647232

http://twitter.com/JetGibbs/status/566802156582608896

http://twitter.com/JetGibbs/status/566802262295846912

So, to recap what I understand as the essential elements from the above:

At the time (2003-2004?), all critical business infrastructure data was on one laptop;
…and it was dying while at an RT convention, so Jet called in her husband’s help.
Jet did not work at the EC HQ, but this was stated (to her) to be the only copy of EC’s data for various critical systems.

Ellora’s Cave Decides to Respond

So Ellora’s Cave’s CEO, Patty Marks, sends a puff piece to their biz group. Which, by the way, happens to be hosted on yahoogroups, exactly where I’d (not) expect a company with a competent IT infrastructure to host critical infrastructure mail groups….

Letter follows.

Sent: Monday, February 16, 2015 2:36 PM
Subject: [ec_biz] Gossip regarding our systems

I don’t like to address gossip, because it lends to the possibility that someone may find it credible, but I think this is important.Apparently there is someone out there saying that we don’t back up your information and that we run our systems on single computers without backup.This so ludicrous that it shouldn’t need addressed, but just in case…

All of our data, including but not limited to Financial, Manuscripts, Graphics, Spreadsheets, programs and any resources used in the daily business of the company past or present is currently stored in a multi-server network.Each server is raided in the event of a hard drive failure and run we run redundant power supplies as an extra precaution.All data stored on the network is then backed up to another location using automatic backup software.The entire room is on its own electrical panel with commercial grade surge protection and battery backup.Our server network and all computers that access it are protected by the latest in antivirus and firewall technology.We have and still do employ a full-time IT department since 2005 – not to mention three outside consulting and hosting companies, one for the website, one for our computers and servers and one for our accounting programs.

Before that, we were a company consisting of six people or less and had no server network at that time.IT services were subcontracted on an as needed basis.Computers were backed up individually to external storage devices and no one computer contained all of the company information – at least not since I have been with Ellora’s Cave.As the company has progressed so has our hardware and software that we use every day to run things.

As for the confusion in the data loss with the Amazon cloud crash, there was no royalty data lost.The information erased were certain formulas that were built into the back end of the old EC site.We had the consultants who designed the original formulas fix the broken code – using the backups – and install it on our servers as a standalone program.All financial spreadsheets, imported or exported, royalty programs hosted offsite or onsite, have always been stored on our server network as well.Again we run like seven server environments for redundancy.

Our email system and our website all run on multiple cloud based servers for backup redundancy.Our own network has its own backup redundancy as described above.All user passwords and master passwords have to meet strict password requirements and are changed on a regular basis.All internal users require a domain credentials to access the files they have been give permissions to view and remote users need their domain credentials as well as firewall credentials to gain access to the internal network.No one person has access to the entire system except the IT department.All other permissions are locked down by department and department level basis.

AND – on top of it all – we carry business insurance in case of hardware or software failure and data loss.

Can I say with certainty that our systems are absolutely impenetrable?Well, considering people have hacked banks, giant retailers, SONY and managed to crash the Amazon cloud, I would guess not.But to imply that all of our information is on one “dying laptop” is laughable and simply untrue.

My Commentary on the Above

It’s generally a convention of the English language to put spaces after periods and before new sentences. I’d expect a publisher to know that.
It wasn’t actually gossip. That’s awfully dismissive.
Anyone who thought it might be a statement about Ellora’s Cave’s current business practices needs better reading comprehension skills.
Methinks EC doth protest too much.
“We have and still do employ a full-time IT department since 2005 […}.” Conveniently, none of these people appear on Ellora’s Cave’s staff list.
“Each server is raided in the event of a hard drive failure[…].” Okay, I actually LOLed at that. I assume Patty means: “Each server has RAID arrays to prevent loss of data from hard drive failure” and not what it says, which is: “We cannibalize servers when hard drives fail.”

My Own Recent Oops

I’ve lost data, and it’s not pretty. In fact, I’ve been considering writing two separate pieces about preventing data loss: one about several simple things you can do to ensure you don’t lose your data on your hard drive, and the other would be ways to ensure you have a full backup of a WordPress site. The latter involves some harder problems, though.

I have what I consider a paranoid level of backups. The other day, I had to wipe my server (that no longer hosts deirdre.net but still hosts several other domains I run).

I bought another domain the other night and was wondering where I was going to put it—my VPS (virtual private server) was out of space. Then I remembered I’d paid for more space, so I should just use that. Except that meant re-partitioning the drive, which meant wiping it, and then I decided I wanted to switch Linux distributions while I was at it. So…been at that for a few days.

I use the utility rsync to push my sites to my VPS. This is an old habit of mine.

Each site is kept in its own git repository. For those unfamiliar with git, it manages source code control. So, essentially, I can get back any revision at any point. Git leaves an invisible directory, .git, at the top of the repository.

Unfortunately, when I pushed one site up, I forgot to exclude the .git directory, and I canceled the transfer in the middle. Remember, I was space-constrained and git repositories aren’t particularly small.

Then I forgot to remove the partial .git directory.

So I’m in a hurry to back up my domains before wiping my VPS and resizing the partition and reinstalling and yada yada yada, and what I meant to do was to copy the wp-content/uploads/ directory from the server back to my own hard drive, so I didn’t lose any photos. As Jenny Trout did. Disclosure: I have made the exact same mistake Jenny did, and that’s why chair-in-the-sky.com doesn’t have any of its old posts. I’ve removed them until I can dig up the photos. Sigh. Like Jenny, I lost them in a site transfer. Of all the steps in moving WordPress installs across sites, the media library is the most fragile part.

Rewinding a bit: I’m in a hurry. Instead of that one directory, I rsync back to my hard drive the whole domain, including the partial .git directory.

Clobbering my repository.

Sad panda.

But!

I have always pushed my git repository to another site! It’s all good, because my backup’s only a few minutes old. I can re-init a git repository and pull from the remote site.

However, the quicker option? I restored a thirty-minutes-ago backup of that directory with Apple’s Time Machine. Then I correctly pull the uploads directory only and add the new contents to my git repository. And push it upstream.

::phew::

To many of you, what I just wrote sounds completely unintelligible, but I do have a point in writing it (apart from personal embarrassment).

I’m a one-person shop, and this is the skill and expertise I have to ensure my own data doesn’t get lost, even when I screw up when I do something I’m too tired and/or rushed to do correctly.

Ellora’s Cave would need that much more skill on the ground, and their staff list doesn’t reflect that.