I broke my 56 hour day this morning and took a nap before coming back.  One of the nice features of Stardock is that it has developers in the US and in Europe so we can, technically, work on things around the clock if necessary which is something we’re taking advantage right now.

So where are we at?

Today

The system works pretty well if you have a few thousand people online at once.  The system works…less well if there are tens of thousands of people online at once. And if there are over 100,000 people, well, you get horrific results such as the game being incredibly unresponsive due to simple web service calls that were considered pretty benign during the beta that suddenly start to bring down firewalls and such due to the sheer massive number of calls that are being made.

Sadly, most of the ~120,000 connections are not customers but via warez.  About 18,000 are legitimate.  So anyway, we spent a lot of time today  trying to isolate out the warez users from the legitimate users (it would require a lot of surgery to actually break them and even if we did, there’d be no friendly “ha ha pirate” message which would result in people just saying the game is buggy).  Mind you, the game makes relatively few server calls, it’s just the sheer number of people.

Our stress tests had counted on having maybe 50,000 people playing at once at peak and that wouldn’t be reached for a few weeks by which time we would have slowly seen things becoming problematic.  With Sins of a Solar Empire, the game was incredibly successful but its user base grew slowly and steadily over time.  Sometimes on a peak time the server would start to get slow and we would adjust so that it would be better next time.

But here, when you’re getting that many connections at once, you’re no longer talking about dealing with the basics like having a good SAN and lots of redundant servers. You instead fall into all kinds of weird secondary issues that start to pop up like yesterday’s case where an older network card couldn’t handle all the packets (not bandwidth limited but # of tiny packets being sent out at once).  

So during the day today, people couldn’t even logon and in some cases, the Demigod forums, which use one of the affected databases for some piddly thing were even down. Even getting the game running was a pain today because a simple HTTP call to see what the latest version would get hung leaving people looking at a black screen. Stuff of nightmares.

Before the game shipped, I wrote a scary email to our team saying how disastrous things would be and predicted doom for us and GPG if there were problems with multiplayer. At the time, my worry was about things like disconnects and CVP. It didn’t occur to me that we’d have near MMO user connections to throw in.

Tonight

Last night and all day today we’ve been working on multiple fronts. 

First, we’ve been trying to shuttle off the warez users so that they’re not slamming our servers. Well, not slamming the same servers that legitimate customers are using.  I’m feeling pretty confident that that is working better because there’s a lot more people in games with lots of people in them.

Second, we’ve developed a new connection system (pictured below).  The connection stuff in Demigod is not part of Demigod but is rather part of Impulse Reactor which in turn is talking to the various connection servers we have set up to get player A to connect to player B and so on.  It’s not done yet but amongst its new features is the ability for the host to eject players who just can’t connect.

This whole new connection service has been written in the past 48 hours.

Third, we’ve greatly expanded our capacity.  I tell you, anyone who wants to know why we delayed Society (MMO RTS we are slowly working on) need look no further than how painful the first 24 hours since the official release of Demigod have been. That was done today and hopefully users have noticed an improvement.

We’re in the process of testing out this new update to put up tonight (less than 24 hours after the street date) that should largely fix this.

Early reviews

The early reviews of Demigod have been very good – in general.

Neoseeker gave us a 10 out of 10.

Crispy Gamer gave us a “Buy It”

But the GameSpot review nailed us because of the connectivity with a 6.5. I’m pretty disappointed about that. [Begin Rant] First, I totally understand that connectivity is central to a game like this. I totally agree. But I think that should be weighted with what the average user who gets Demigod will experience and in reality, as annoying as this issue is, it’s not something that’s going to be an ongoing issue, it’s something that is likely to be taken care of in the next day or two.  So this time next week, players will be happily playing but GameSpot’s review will live on.  I suppose the solution going forward isn’t to send out review copies until the game is released and we make sure there’s no network issues. Afterall, in a week from now, the guy buying it from retail will have a great experience. [End Rant]

The IGN and 1Up reviews are coming up.  So we could be looking at a lot of reviews like that if every reviewer decides to take the same approach Gamespot did which I hope they don’t (luckily for every other on-line centric game that’s had a rough first day or two most reviewers haven’t taken Gamespot’s approach).

Video

We took a video about some of the stuff we’re working on and a little quick tour of Stardock. You can see it here:

IGN Video