Category Archives: Software

Source Control by Shingle

While reading about “The Stalled Server Room” over at The Daily WTF I found this other great entry on the source control by shingle model. Apparently some guy, so afraid of using any tools to aid in his development, used an actual shingle in order to manage who had the right to modify the base libraries in the project.

Read the whole story titled “The Source Control Shingle“, it’s hilarious. Surprise, surprise, they are developing on Windows! 🙂

Synchronized variable access in C++

I’m working with an API for a closed source library that comes with a recommendation of using two separate threads when passing it data in order to ensure that the data pipelines are never empty. I’ve been doing it for a while but recently we had a lot of stability issues. After some digging I tried to rewrite the two threads into one that multiplexed the calls instead and suddenly stability was back again. I think that if you recommend that people access your API via threads it would be nice to indicate that some calls aren’t thread safe, but no.

I decided to design a generic type that would allow synchronized access to any kind of variable while utilizing RAII to ensure that it is unlocked automatically. The task turned out to be a little more complicated than I first expected but after some trial and error I came up with a solution that I found had a good functionality/aesthetic ratio.

template <typename T>
class SynchronizedVariable : public boost::non_copyable {
public:
    class Accessor {
    public:
        T* operator->() {
            return m_variable;
        }

        const T* operator->() const {
            return m_variable;
        }

    private:
        Accessor(const boost::recursive_mutex::scoped_lock& lock, T* variable) :
            m_lock(lock),
            m_variable(variable) { }

        const boost::recursive_mutex::scoped_lock& m_lock;
        T* m_variable;

        friend class SynchronizedVariable<T>;
    };

    SynchronizedVariable(const T& variable) :
        m_mutex(),
        m_variable(variable) { }

    SynchronizedVariable() :
        m_mutex(),
        m_variable() { }

    const Accessor get() const {
        return Accessor(m_mutex, &m_variable);
    }

    Accessor get() {
        return Accessor(m_mutex, &m_variable);
    }

private:
    mutable boost::recursive_mutex m_mutex;
    T m_variable;
};

SynchronizedVariable is the actual container for the variable and you can either pass it a constant reference to an existing instance which will be copied or create a new empty instance. The idea is that you should not be able to retain an external copy which can be accessed unprotected. It is of course possible to access the original instance if the variable is copied but this is a limited case of programming error.

Subsequent accesses to the variable must go through an instance of the SynchronizedVariable<T>::Accessor object which ensures that the variable is protected by a mutex and that it is automatically unlocked when the accessor goes out of scope. It is of course possible to add lock/unlock methods to it so that the user can limit the scope of the lock. Personally I try to limit all my scopes as much as possible, especially in areas of code which is executed frequently and during performance sensitive moments.

Accessor is storing the variable as a pointer rather than a reference. The reasoning behind this is that operator-> has to return a pointer so storing a reference does not make a lot of sense. There is also a lot of mutex locking/unlocking going on if you instantiate the accessor in frequently called small scopes so be prepared to dust off gprof if you suspect that this is the case.

Python; impressively portable

Last year I was faced with the daunting task of converting a large amount of Bash scripts to C-code. A terrible thing to do since we did a lot of stdio processing/redirecting which is simple to do in Bash but horrible in C. The reason for this atrocity was the fact that Bash did not work on our uClinux platform and none of the clones implemented enough feature to be compatible with our rather complicated scripts.

I went with a multicall binary in order to reduce the flash storage footprint and the end result is 100% compatible with the original scripts. On the downside the SLOC increased nearly four times and it was a PITA to implement.

Recently Kristian recommended that I should take a look at Python. At first I was skeptical for several reasons. If Bash wasn’t portable to uClinux then why would Python be considering it has so many more features. I’m also very careful about adding large complex software packages to an already sensitive platform.

A couple of weeks ago I had some time over and was terribly bored so I decided to give cross-compiling Python a go. As it turns out their build system isn’t cross-compile-aware at all but I did manage to build a working interpreter. Today Kristian found a guide on how to cross-compile the modules as well and so we were able to build a complete Python distribution. Surprisingly it seems to be working very well and so far memory consumption is within reasonable boundaries which is impressive. I’m not sure I want to port all the already converted Bash-scripts to Python right now but at least it is a perfectly viable option and would most likely increase code quality significantly in the long run.

Available socket bytes

Ever wondered how to find out how many bytes can be read from a socket? I did and it is really simple although possibly not portable.

ssize_t socket_get_available(int socket_fd)
{
        int available = -1;
        assert(0 <= socket_fd);

        if (ioctl(socket_fd, FIONREAD, &available) < 0) {
                return -1;
        }

        return (ssize_t) available;
}

OpenGL 3.0, flip/flop

Ever since I released the AdaTerrainEngine source code I’ve started to get an itch to go back into the OpenGL programming game again.

Something I’ve always been curious about is if Direct3D does have any big advantages over OpenGL but finding a good answer to that question has been hard, probably because there isn’t any. There is no doubt about OpenGL being much more portable and I’ve heard from good sources that it is much easier to get into OpenGL programming than Direct3D. For instance, you need approximately the same amount of lines of C/C++ code to set up an OpenGL window with SDL that you need to write a Hello World! application.

Recently someone told me that Microsoft’s latest DirectX release, version 10, is only available for Vista. I don’t find that weird considering the company’s history of forcing users to upgrade to something they don’t need however what I did enjoy was the fact that Crysis is (somewhat) playable with Wine. This indicates that you can recreate if not all at least the required parts of Direct3D 10 using OpenGL. Microsoft’s excuse for only delivering DirectX 10 to Vista is that it requires the new driver API they invented. I guess that means X11/OpenGL did it right the first time then, hooray.

Peter Kirn argues that OpenGL 2/2.1 offers the same functionality as DirectX 10 without the obvious vendor lock-in. Sometimes you hear people saying that since DirectX is updated more frequent than OpenGL it must be the case that it supports the latest GPU features whereas OpenGL does not. This is very simply not true due to the fact that the base OpenGL API only implements common and accepted functionality available across most GPUs. New features that have just been invented by someone are implemented as OpenGL extensions. These extensions can be implemented at any time and really only requires you to have the latest vendor-supplied driver which will support it. The effect of this is actually that OpenGL can support these new features quicker than DirectX since they do not have to release an entirely new version of their API! One such example is geometry shaders which are implemented by Direct3D 10 but available as GL_EXT_geometry_shader4 in OpenGL 2.1.

The fact is that OpenGL 3.0 is well on its way and it will offer an entirely rethought API which will much better meet todays needs. One of the biggies being that is more object-centric (whereas the current implementation is based on a finite state machine. The API will still be C compatible!

Another big advantage, according to “Some Thoughts on OpenGL 3.0“, is that you no longer need to tediously have to input every single vertex. Instead you can now “stream” a large amount of vertices into OpenGL making it much easier to load large, complicated, models. Another interesting feature is that everything in the API which can be created by shaders will be removed, leading to a cleaner more minimalistic API as opposed to DirectX which is constantly growing.

As desktop adoption of Linux and Mac OS X is constantly growing I believe that OpenGL will regain most of its fame soon. OpenGL 3 needs to be heavily marketed when it is released as I suspect many game developers have been listening to Microsoft for a long time now and they might need some encouragement to take the plunge. Writing portable games does not need to be something difficult as long as people are using the right tools for the job and the gaming industry need to wake up and smell platform diversity explosion that will soon be upon us. The only people I know that still argue against OS X on the desktop are the very same people that have never tried it. As for Linux, it tailors to the more computer literate and in my opinion it should stay that way. Still, the number of computer literate people should be enough to make it an interesting platform for entertainment providers.
nVidia gave a short OpenGL 3 presentation at SIGGRAPH 2007 which is available here.

axTLS, embedded security

Recently we were faced with the fact that our current encryption model isn’t strong enough to satisfy some of our more demanding customers. After discussing the possibilities of writing our own encryption model based on public strong encryption standards we finally decided to go for a SSL solution. Building upon an existing and proven standard carries much more weight than reinventing the wheel and I was faced with the task of locating a suitable SSL implementation for our embedded systems.

The first two implementations that popped into my head were of course GNU TLS and OpenSSL. After investigating both solutions I came to the following conclusions. GNU TLS has some external dependencies, such as libgcrypt and libgpg-error. As I want to keep dependencies to a minimum since we integrate them with our build system, which can be a painful process, that sort of spoke against GNU TLS. Though OpenSSL didn’t have any external dependencies we couldn’t already satisfy it had a problem which also plagued GNU TLS, it had a rather large footprint. Somewhere in the vicinity of five times larger than the binary it was going to be linked with. This was clearly something that would be difficult to motivate.

After some googling I discovered three very interesting alternatives, MatrixSSL, XySSL and axTLS. All designed for embedded systems and having an acceptable footprint.
MatrixSSL costs money for commercial use so I decided to push it to the back of the queue. XySSL had some weird problems on our platform resulting in the infamous “TANGO15 reboot“. axTLS, while only supporting TLSv1, worked flawlessly, was easy to compile and had a footprint of roughly 50% of the binary it would be linked with. Performance isn’t a factor for the application in questions and hence it was not investigated.

If you are looking for an SSL implementation for an embedded system and TLSv1 is good enough for you I can highly recommend axTLS.

Simple Buffer Parsing With C99

A friend of mine asked some questions relating to socket programming which spurred me to create a simple example based on an application I wrote a while back. It’s pretty basic stuff but I decided to post it here in case anyone has any feedback to offer.

I use flexible arrays where suitable so warm up your C99-compilers. There is virtually no error checking done in the example so be vary, also I didn’t compile it. 😉

static const uint32_t PROTOCOL_MAGIC=0xdead001;
struct HelloHeader {
    uint32_t magic;
    uint32_t version;
};

enum Command {
    Command_NONE,
    Command_FILE_LIST,
    Command_GET_FILE,
};

struct CmdHeader {
    uint32_t cmd;
};

struct File {
    char name[256];
    uint64_t size;
};

struct FileList {
    uint32_t numFiles;
    File[] files;
};

struct GetFile {
    uint64_t len;
    /* null-terminated filename followed by data */
    char[256] name;
    char[] data;
};

uint64_t ntohll(uint64_t n)
{
#ifdef BIG_ENDIAN
    return n;
#else
    return (((uint64_t) ntohl(n)) << 32) + ntohl(n >> 32);
#endif
}

char* processCommand000301(char* buffer, size_t bufferLen)
{
    /* ... */
}

char* processCommand010000(char* buffer, size_t bufferLen)
{
    char* ptr = buffer;

    struct CmdHeader* commandHeader = (struct CmdHeader*) ptr;
    commandHeader->cmd = ntohl(commandHeader->cmd);
    ptr += sizeof(struct CmdHeader);
    switch (commandHeader->cmd) {
        case Command_FILE_LIST: {
            struct FileList* fileList = (struct FileList*) ptr;
            fileList->numFiles = ntohl(fileList->numFiles);
            ptr += sizeof(struct FileList);
            for (int i = 0; i < fileList.numFiles; ++i) {
                fileList.files[i].size = ntohll(fileList.files[i].size);
                printf("%s %lld\n", fileList.files[i].name, fileList.files[i].size);
                ptr += sizeof(struct File);
            }
            break;
        }

        case Command_GET_FILE: {
            struct GetFile* getFile = (struct GetFile*) ptr;
            getFile->len = ntohll(getFile->len);
            FILE* file = fopen(getFile.name, "w");
            fwrite(ptr, 1, getFile.len, file);
            ptr += getFile.len;
            fclose(file);
            break;
        }

        default:
            /* Unrecognized command! */
            return NULL;
    }

    return ptr;
}

int processBuffer(char* buffer, size_t bufferLen)
{
    char* ptr = buffer;

    struct HelloHeader* helloHeader = (struct HelloHeader*) buffer;
    helloHeader->magic = ntohl(helloHeader->magic);
    helloHeader->version = ntohl(helloHeader->version);
    if (helloHeader->magic != PROTOCOL_MAGIC) {
        /* Invalid header! */
        return -1;
    }
    ptr += sizeof(struct HelloHeader);

    while (ptr < (buffer + bufferLen)) {
        switch (helloHeader->version) {
            case 0x00010000:
                /* version 1.0.0 */
                ptr = processCommand010000(buffer, (size_t) ptr - buffer);
                break;

            case 0x00000301:
                /* version 0.3.1 */
                ptr = processCommand000301(buffer, (size_t) ptr - buffer);
                break;

            default:
                /* Unsupported version! */
                return -1;
        }
    }
    return 0;
}

P.S. I’m impressed by WordPress’ ability to consistently botch preformatted text when using the WYSIWYG editor. D.S.

Terrain rendering in Ada

A while ago I added another old project of mine to the projects page but I neglected to post about it.

The project is based on an old school project of mine from the university. It’s a terrain rendering engine written in Ada which was intended to be capable of handling very large terrains of very high detail, +/- 1 meter at the maximum zoom level if I recall correctly.

Since at the time the few OpenGL-bindings that existed for Ada were highly outdated and of varying quality I quickly decided I had to remedy the situation myself. It resulted in the AdaOpenGL project which to this very day, even though I haven’t updated it in over four years, still has a significant amount of downloads (according to SourceForge statistics). People have even tracked me down and called me personally about it, which was a first for me. For these very reasons I often have a guilty conscience for not bringing the project up to date. Two developers have offered to assist but neither of them every produced any releases.

After I had a viable OpenGL-binding I started on the terrain engine itself. As you might suspect I spent a lot of time on the binding and based on the little time I had left I decided to only implement a basic scene graph for the terrain engine. The idea was that someone else would pick up where I left off but as it turned out the company I did the project with never asked for my source code, which I found rather weird considering their initial enthusiasm.

I gave the project the name AdaTerrainEngine as I cannot recall what I initially named it. The data files belong to the company which I did the project with so I cannot release them. I don’t really remember how useful the source code is but at least it should provide a basic foundation for anyone who is doing OpenGL work in Ada.
The terrain engine is LGPL-licensed and AdaOpenGL is BSD-licensed.

P.S. I blatantly stole the design for the AdaOpenGL-site from an awesome CSS tutorial. After I finished the page I realized I forgot to credit the original author and I couldn’t find the site again. If you have any idea who did this please let me know! D.S.