Translations!

flattr this!

The translation project has been ongoing for quite some time, and so far it has resulted in a finished Norwegian translation and a nearly Finnish one. If you’re interested in these, you can download the files and replace the files in gamedata\uitext\english.dir with them.

The Google Code repository also contains an installer for Norwegian, but it requires you to configure language using the client’s app.config file.

For information on how to help out, please see this.

Authentication and encryption

flattr this!

Lately I’ve been working on safe authentication and encryption for Project Dollhouse’s network library, GonzoNet. After a lot of deliberation, the choice fell on Elliptic Curve Diffie Hellman (ECDH) for authentication and AES for encryption.

ECDH is relatively simple to implement, and is safe enough for the purposes of a game. It also makes it simple to establish encryption parameters net ieded for further encryption of a protocol.

In other news, we’re making great progress on the Simantics VM, so that the only things remaining are network related features.

For an example of how GonzoNet implements ECDH, and how you can use it in your own projects, look here!

KISS

flattr this!

KISS is a well known acronym in design circuits. According to Wikipedia, it was first noted by the U.S Navy in the sixties.

When I was going for my second attempt at creating a fully fledged auto updating framework, this acronym flashed before my eyes before I’d written as much as a single line of code. The last one I did was over designed, and I could never get it to work the way I wanted it to.

The main way in which it was over designed was that it kept a record of previous software versions/patches. This was stupid. It required too much bookkeeping, for absolutely no gain. By the time new patches are deployed, any information in a previous patch is outdated anyway.

Therefore, I took it upon myself to do it, this time, the Right Way (to paraphrase John Carmack.)

The first vital decision I made was to put the actual downloading code into a library, so that it could be separated from the application itself and be reused. Obviously, I named the library KISS.

The second vital decision I made was to ignore any notion of previous versions/patches. This resulted in the following:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.Net.Security;
using System.IO;
using System.Threading;
using System.Security.Cryptography.X509Certificates;

namespace KISS
{
    public delegate void FetchedManifestDelegate(ManifestFile Manifest);
    public delegate void FetchedFileDelegate(MemoryStream FileStream);
    public delegate void DownloadTickDelegate(RequestState State);

    /// <summary>
    /// A requester requests and fetches files from a webserver, and calls
    /// events when the requests are complete.
    /// </summary>
    public class Requester
    {
        //This is implementation specific. Personally, I'm ignoring security to avoid headaches.
        public static bool ACCEPT_ALL_CERTIFICATES = true;

        public event FetchedManifestDelegate OnFetchedManifest;
        private string m_ManifestAddress = "";

        /// <summary>
        /// Contains information about a download in progress,
        /// such as: percent complete and KB/sec.
        /// </summary>
        public event DownloadTickDelegate OnTick;

        /// <summary>
        /// Called when a file was fetched!
        /// </summary>
        public event FetchedFileDelegate OnFetchedFile;

        private bool m_HasFetchedManifest = false;

        public Requester(string ManifestAddress)
        {
            m_ManifestAddress = ManifestAddress;
        }

        public void Initialize()
        {
            try
            {
                WebRequest Request = WebRequest.Create(m_ManifestAddress);
                RequestState ReqState = new RequestState();

                ReqState.Request = Request;
                ReqState.TransferStart = DateTime.Now;
                Request.BeginGetResponse(new AsyncCallback(GotInitialResponse), ReqState);
                ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(AcceptAllCertifications);
            }
            catch (Exception E)
            {
                Logger.Log("Exception in Requester.Initialize:\n" + E.ToString(), LogLevel.error);
            }
        }

        /// <summary>
        /// Starts fetching a file, and notifies the OnFetchedFile event
        /// when done.
        /// </summary>
        public void FetchFile(string URL)
        {
            WebRequest Request = WebRequest.Create(URL);
            Request.Method = "GET";
            RequestState ReqState = new RequestState();

            ReqState.Request = Request;
            ReqState.TransferStart = DateTime.Now;
            Request.BeginGetResponse(new AsyncCallback(GotInitialResponse), ReqState);
            ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(AcceptAllCertifications);
        }

        private void GotInitialResponse(IAsyncResult AResult)
        {
            RequestState ReqState = (RequestState)AResult.AsyncState;
            ReqState.Response = ReqState.Request.EndGetResponse(AResult);
            ReqState.ContentType = ReqState.Response.ContentType;
            ReqState.ContentLength = (int)ReqState.Response.ContentLength;

            Stream ResponseStream = ReqState.Response.GetResponseStream();
            ReqState.ResponseStream = ResponseStream;
            ReqState.RequestBuffer = new byte[ReqState.ContentLength];
            ResponseStream.BeginRead(ReqState.RequestBuffer, 0, (int)ReqState.Response.ContentLength,
                new AsyncCallback(ReadCallback), ReqState);
        }

        private void ReadCallback(IAsyncResult AResult)
        {
            RequestState ReqState = ((RequestState)(AResult.AsyncState));

            Stream ResponseStream = ReqState.ResponseStream;

            // Get results of read operation
            int BytesRead = ResponseStream.EndRead(AResult);

            // Got some data, need to read more
            if (BytesRead > 0)
            {
                // Report some progress, including total # bytes read, % complete, and transfer rate
                ReqState.BytesRead += BytesRead;
                ReqState.PctComplete = ((double)ReqState.BytesRead / (double)ReqState.ContentLength) * 100.0f;

                // Note: bytesRead/totalMS is in bytes/ms. Convert to kb/sec.
                TimeSpan totalTime = DateTime.Now - ReqState.TransferStart;
                ReqState.KBPerSec = (ReqState.BytesRead * 1000.0f) / (totalTime.TotalMilliseconds * 1024.0f);

                OnTick(ReqState);

                // Kick off another read
                IAsyncResult ar = ResponseStream.BeginRead(ReqState.RequestBuffer, ReqState.BytesRead,
                    (ReqState.RequestBuffer.Length - ReqState.BytesRead), new AsyncCallback(ReadCallback), ReqState);
                return;
            }

            // EndRead returned 0, so no more data to be read
            else
            {
                ResponseStream.Close();
                ReqState.Response.Close();
                ReqState.ResponseStream.Close();
                OnFinishedFile(new MemoryStream(ReqState.RequestBuffer));
            }
        }

        /// <summary>
        /// Finished downloading a file!
        /// </summary>
        /// <param name="FileStr">The stream of the file that was downloaded.</param>
        private void OnFinishedFile(MemoryStream FileStr)
        {
            if (!m_HasFetchedManifest)
            {
                m_HasFetchedManifest = true;
                OnFetchedManifest(new ManifestFile(FileStr));
            }
            else
            {
                OnFetchedFile(FileStr);
            }
        }

        private bool AcceptAllCertifications(object sender, X509Certificate certification, X509Chain chain, SslPolicyErrors sslPolicyErrors)
        {
            return ACCEPT_ALL_CERTIFICATES;
        }
    }
}

This is just sexy. It is short, to the point, easy to maintain, flexible and elegant. I will eventually port this to C++.

To explain, the KISS framework does most of its work based on a local and remote manifest file, which can be generated by a tool called Manifestation. A manifest file simply contains the version of a patch (or the client’s current version, if the manifest is a local one), the number of files contained in the manifest, and then a listing of the files that make up the patch. A file is defined as:

  • Address – the local address of the file, starting from the root folder of the client.
  • Hash – Hash of a file, to determine if the file has changed (and thus, if it needs to be downloaded).
  • URL – The URL of the file.

The actual code is equally simple:

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace KISS
{
    /// <summary>
    /// A manifest file is a file that has a version and references a bunch of patch files.
    /// </summary>
    public class ManifestFile
    {
        public string Version = "";
        public List<PatchFile> PatchFiles = new List<PatchFile>();

        public ManifestFile(string Path, string Version, List<PatchFile> PatchFiles)
        {
            bool HasURLs = false;
            BinaryWriter Writer = new BinaryWriter(File.Create(Path));
            Writer.Write((string)Version);

            if (PatchFiles[0].URL != "")
                HasURLs = true;

            foreach (PatchFile PFile in PatchFiles)
            {
                if (!HasURLs)
                    Writer.Write((string)PFile.Address + "," + PFile.FileHash);
                else
                    Writer.Write((string)PFile.Address + "," + PFile.FileHash + PFile.URL);
            }

            Writer.Flush();
            Writer.Close();
        }

        /// <summary>
        /// Creates a ManifestFile instance from a downloaded stream.
        /// </summary>
        /// <param name="ManifestStream"></param>
        public ManifestFile(Stream ManifestStream)
        {
            BinaryReader Reader = new BinaryReader(ManifestStream);
            Reader.BaseStream.Position = 0; //IMPORTANT!

            Version = Reader.ReadString();
            int NumFiles = Reader.ReadInt32();

            for(int i = 0; i < NumFiles; i++)
            {
                string PatchFileStr = Reader.ReadString();
                string[] SplitPatchFileStr = PatchFileStr.Split(",".ToCharArray());

                PatchFiles.Add(new PatchFile()
                {
                    Address = SplitPatchFileStr[0],
                    FileHash = SplitPatchFileStr[1],
                    URL = SplitPatchFileStr[2]
                });
            }

            Reader.Close();
        }
    }
}

This has but one disadvantage: A hash is stored as text, so that comparing them isn’t particularly fast, but this can obviously be changed to suit your need if you decide to use the framework. My plan is to eventually port the framework to C++; for now, it is available as C# code at Github. The license is Mozilla 2.0 – my personal favorite. If you change anything, you have to share it. Other than that, use as you please!

Going live

flattr this!

As I’m writing this, I’m at Oslo Airport Gardermoen, waiting for a plane to take me back to Bodø.

Technically speaking, Kon Tiki was considered done a couple of days ago. I have procured a server at Accuwebhosting, and the Loginserver is currently running.

However, I am unable to log in with the client, most likely because of an issue known as NAT traversal. What this means is that packets sent from a client behind a router need to find their way to the server (which is also typically behind a router) and vice versa.

It is possible to do this manually by forwarding ports in one’s router, but I am trying to avoid this by programmatically forwarding ports. This means more code and a few more days of waiting, but everything good shall come to those who wait!

Kon Tiki almost done

flattr this!

Last night I released the client – it can be downloaded here. I should start by apologizing for not being able to keep my promise. The main reason for this can basically be summed up in one word: Christmas.

Anyway, the client is now considered feature complete for this milestone, and any updates to it from now on will only be to fix bugs.

Though we’ve spent a long time, we’ve managed to get further than originally anticipated; the client now supports not only character creation, but logging into a city as well.

I’m hoping to go live on or before New Year’s Eve, but I’m making no promises this time. At this point, it is prudent to give thanks to ddfczm, without whom the client would never have looked as good as it does. At least not in the same timeframe!

Andrew D’Addessio, alias Fatbag, also deserves a big thanks for the revolutionary and groundbreaking work he’s done on R&D (Research and Development). Without him, there would be no Project Dollhouse!

Character creation done

flattr this!

First character created in the database. Ever.

First character created in the database. Ever.

Character creation is now done. It went surprisingly smooth, taking only a few hours. If transitioning to the CityServer goes as smoothly, the first milestone will be done within the next week.

From there on out, what remains is to go live with the servers, which is an issue in an of itself, as it requires a lot of configuration and general messing about. But the first milestone will go live before Christmas, that is a promise!

Back on track!

flattr this!

So, after a prolonged setback, I finally managed to compile the client using XNA 4. That means that development is now back on track, and I’m going to be debugging some network issues with the client before attempting to pull everything together to a release! :D
The neat thing about this is that we probably (hopefully) have a working Monogame client as well, but I haven’t dared to try to run it yet, due to Monogame’s inherently unstable nature.

Windows 8 rant

flattr this!

Seems like the Kon Tiki milestone might have to be postponed till the end of August.

I’m currently trying to get Visual Studio 2008 installed on my laptop, and it is proving to be an incredibly frustrating experience. This leads me to regret that we haven’t ported the entire codebase to Monogame by now, as well as regretting the decision to install Win8 on the laptop.

It started with error 1935, which apparently is a catch all errorcode for unspecified problems during installation. However, when you get an hresult of 0x80073712, it means that your component store is corrupted.

So far, so bad. If only this problem could be fixed. But it turns out, it can’t. Not readily, anyway. I’ve tried running DISM with /scanhealth and /restorehealth – no luck (aside from confirming what I already knew – component store corruption).

I then tried running the Windows Update Diagnostic tool, which fixed something… but VS 2008 still won’t install. It fails at pretty much exactly the same place, so I haven’t even bothered checking the logs to see if the errorcode changed.

I’ve also run CCleaner to fix any potential registry issues, still no go. If anyone have had any similar experiences, please let me know below!

Argh, Microsoft!

Packet dimensions

flattr this!

Development has been paused for a while, mostly due to everyone being busy doing other stuff. However, now we are getting back into the swing of things.

Yesterday, I discovered that for some reason, I had decided that it was a good idea to encode a packet’s length in a single byte when I first implemented the game’s networking code.

Clearly, it isn’t. Unless you want to start packing bits to save space (which quite frankly isn’t worth the time and effort), a byte can only represent numbers up to 255. That means packets could only be 255 bytes long. This isn’t very effective, or even practical.

Therefore I decided to use an unsigned short (2 bytes) to encode a packet’s length, and another to encode a packet’s decrypted length (if it is encrypted). This means packets can now be up to 65,025 bytes in length. If I find myself needing to send packets any larger than this, I know I need to redesign the protocol.

Of course, doing this made sure that decoding packets received from the client fucked up. There’s always something. But at least development is back on track.

 

Forum

flattr this!

We now have a forum, available here. For the time being, it is mostly in place to help you help us by posting images you have of TSO, so that we can recreate it accurately. We are especially interested in images of the city screen.

We’ll also use it to post advertisements about positions in the team. Currently, we’re looking for a skilled C# developer with access to a Mac or Linux machine that is willing to port the codebase to Monogame.

These forums will not be heavily moderated, and our involvement with the community will be limited because of time constraints (we’re doing this in our spare time, after all), so please behave nicely!

Kon Tiki Milestone

flattr this!

Last night, a defining moment in this project’s history was reached. We finally made Vitaboy, the character rendering system for the game, work, as shown in the below screenshot.

This means that the Kon Tiki milestone is 83% done, and give or take another month of development, we will have released the first running version of Project Dollhouse. It won’t have any gameplay, but you’ll be able to create a character and store it on our server.

We’ve decided to go with aws for our server solution, so by the time the first client is released, you can log onto a development server hosted by us.

After 1 year, 6 months and 29 days of development, we are finally nearing land. As I am writing this, I’m sitting atop Kon Tikis mast, seeing the first birds fly by after over a year at sea.

Also, flattr buttons are now working, so feel free to flattr away!

The entire team would finally like to extend our heartfelt condolances to all families affected by the Boston bombing.

Flattr and OpenMW

flattr this!

Since people have, over time, requested, and requested again, that we provide a way for them to donate to the project, it has finally happened!

By clicking the Flattr button at the bottom of this post, you can donate a small amount towards server costs,

FInally, I’d like to make a shoutout to OpenMW, which is a project aiming to rebuild Morrowind with open source code. Please check it out!

Getting our priorities in order

flattr this!

Yesterday, a design meeting was held on Skype.

Many things emerged – we started a technical design document, amongst other things. But among the more important things, we decided to get our priorities in order. The UI related code will be refactored, and then all manpower will be directed towards getting Vitaboy to work the way it should (we are currently only having issues with the skeleton).

When that’s done, we’ll have completed the login screen, Select A Sim and Create A Sim. That means we can finally move on to the neighbourhood and the actual gameplay. We haven’t yet put a deadline over our heads, but a 4-month perspective was discussed.

In other news, I finished a book. So if you need something to read while waiting for the game to finish, you can get the ebook here, and a paperback here.

Here’s a little preview for people who are following this blog!

City rendering, Monogame and new teammember

flattr this!

A new year is upon us, and we have some great news!

The team has gained a new member, who has already proven his worth by doing what me and Andrew from NIOTSO had put aside because we considered it too difficult a challenge to tackle at the moment.

Check out these wonderful pictures:

Of course, city rendering isn’t done yet. Stuff like foliage, roads and actual houses still needs to be added, but this is a huge step forward!

I can also confirm that porting to Monogame is going proceeding to plan. There are bumps in the road, as always, but for the most part, Monogame appears to be quite stable and appears to have all of the features that are needed for this game. The thought of being able to provide crossplatform support is a thrilling one, especially when Microsoft’s new OS, Windows Blue, will apparently not support any apps targeted specifically for Windows 8.

Fuck Microsoft, long live Project Dollhouse!

Loading assets

flattr this!

Right, I’ve been working on loading assets for a while now. Preloading (caching) assets was very easy to do – the problem was that the original system for loading assets had flaws in it.

It relied on using the outdated packingslips.dat archive for figuring out where assets were located, which was causing issues with files that had complicated paths.

Therefore I’m now working on a tool, Mr. Shipper, that can generate XML files for the assets that are currently being used, as well as corresponding C# files with structures containing the IDs for these assets.

The process was going swimmingly, but then we were once again stumped by Maxis’ unique development practices. We assumed that since an unsigned integer can hold 4,294,967,295 unique values, it would be enough to map each asset in the game. And it would! The game contains a little over 35,000 assets.

That is but a few percent of the total amount of assets one would need to fill a uint’s total range. But somehow, the geniouses at Maxis decided to reuse FileIDs for different files, so that a FileID has to be combined with its TypeID (which is also an unsigned integer) to be unique. That means that to map an asset to a unique ID in the game, you need an unsigned long, which is 8 bytes rather than 4. In other words, each ID for each asset takes twice as much memory for no particular reason at all!

Monogame?

flattr this!

Right, I haven’t made a new post in a while. Don’t fret though, development is still going strong!

Right now, we are working on fixing the loading system. Assets are now cached, which means loading should be much faster and smoother.

Development on the new phone didn’t go according to plan, so was scrapped.

Now, to the real point of this post: In Windows 8, XNA is dead. XNA apps can still run in Windows 8, which is good. This game will most likely never reach any portable devices anyway. However, the fact that the Metro system in Windows 8 doesn’t support XNA natively makes me worry about the future (looking ahead to Windows 9).

I’m going to install Windows 8 on my laptop, which rules out development for me on that. Therefore, I’m thinking of switching to MonoGame. I already tested and confirmed that it is possible to convert the codebase to XNA 4, and since MonoGame is essentially XNA 4 reproduced from the ground up using Metro-compatible code, it should be ideal for my purposes.

I’ll keep you posted!

 

Rendering on Android

flattr this!

Because I still haven’t really made any progress on rendering 3D with XNA, and just got a new phone (HTC Wildfire S), I’ve decided to branch out technology development to Android using OpenGL ES (Embeded Systems). It is a somewhat unlikely branch (because the likelyhood of Project Dollhouse being ported to Android is slim to none), but my hope is that I’ll be able to learn something that I can apply to XNA.

So far all I have is a spinning triangle using a texture from TSO, but I haven’t really gotten started on the programming yet because I spent quite a while setting up the development environment. I’ll update this post with pictures as soon as I have something interesting to show!

Afr0 Games is now also on Twitter! Come follow us!