13 May 08 Python; impressively portable

Last year I was faced with the daunting task of converting a large amount of Bash scripts to C-code. A terrible thing to do since we did a lot of stdio processing/redirecting which is simple to do in Bash but horrible in C. The reason for this atrocity was the fact that Bash did not work on our uClinux platform and none of the clones implemented enough feature to be compatible with our rather complicated scripts.

I went with a multicall binary in order to reduce the flash storage footprint and the end result is 100% compatible with the original scripts. On the downside the SLOC increased nearly four times and it was a PITA to implement.

Recently Kristian recommended that I should take a look at Python. At first I was skeptical for several reasons. If Bash wasn’t portable to uClinux then why would Python be considering it has so many more features. I’m also very careful about adding large complex software packages to an already sensitive platform.

A couple of weeks ago I had some time over and was terribly bored so I decided to give cross-compiling Python a go. As it turns out their build system isn’t cross-compile-aware at all but I did manage to build a working interpreter. Today Kristian found a guide on how to cross-compile the modules as well and so we were able to build a complete Python distribution. Surprisingly it seems to be working very well and so far memory consumption is within reasonable boundaries which is impressive. I’m not sure I want to port all the already converted Bash-scripts to Python right now but at least it is a perfectly viable option and would most likely increase code quality significantly in the long run.

13 May 08 Available socket bytes

Ever wondered how to find out how many bytes can be read from a socket? I did and it is really simple although possibly not portable.

ssize_t socket_get_available(int socket_fd)
{
        int available = -1;
        assert(0 <= socket_fd);

        if (ioctl(socket_fd, FIONREAD, &available) < 0) {
                return -1;
        }

        return (ssize_t) available;
}

12 May 08 OpenGL 3.0, flip/flop

Ever since I released the AdaTerrainEngine source code I’ve started to get an itch to go back into the OpenGL programming game again.

Something I’ve always been curious about is if Direct3D does have any big advantages over OpenGL but finding a good answer to that question has been hard, probably because there isn’t any. There is no doubt about OpenGL being much more portable and I’ve heard from good sources that it is much easier to get into OpenGL programming than Direct3D. For instance, you need approximately the same amount of lines of C/C++ code to set up an OpenGL window with SDL that you need to write a Hello World! application.

Recently someone told me that Microsoft’s latest DirectX release, version 10, is only available for Vista. I don’t find that weird considering the company’s history of forcing users to upgrade to something they don’t need however what I did enjoy was the fact that Crysis is (somewhat) playable with Wine. This indicates that you can recreate if not all at least the required parts of Direct3D 10 using OpenGL. Microsoft’s excuse for only delivering DirectX 10 to Vista is that it requires the new driver API they invented. I guess that means X11/OpenGL did it right the first time then, hooray.

Peter Kirn argues that OpenGL 2/2.1 offers the same functionality as DirectX 10 without the obvious vendor lock-in. Sometimes you hear people saying that since DirectX is updated more frequent than OpenGL it must be the case that it supports the latest GPU features whereas OpenGL does not. This is very simply not true due to the fact that the base OpenGL API only implements common and accepted functionality available across most GPUs. New features that have just been invented by someone are implemented as OpenGL extensions. These extensions can be implemented at any time and really only requires you to have the latest vendor-supplied driver which will support it. The effect of this is actually that OpenGL can support these new features quicker than DirectX since they do not have to release an entirely new version of their API! One such example is geometry shaders which are implemented by Direct3D 10 but available as GL_EXT_geometry_shader4 in OpenGL 2.1.

The fact is that OpenGL 3.0 is well on its way and it will offer an entirely rethought API which will much better meet todays needs. One of the biggies being that is more object-centric (whereas the current implementation is based on a finite state machine. The API will still be C compatible!

Another big advantage, according to “Some Thoughts on OpenGL 3.0“, is that you no longer need to tediously have to input every single vertex. Instead you can now “stream” a large amount of vertices into OpenGL making it much easier to load large, complicated, models. Another interesting feature is that everything in the API which can be created by shaders will be removed, leading to a cleaner more minimalistic API as opposed to DirectX which is constantly growing.

As desktop adoption of Linux and Mac OS X is constantly growing I believe that OpenGL will regain most of its fame soon. OpenGL 3 needs to be heavily marketed when it is released as I suspect many game developers have been listening to Microsoft for a long time now and they might need some encouragement to take the plunge. Writing portable games does not need to be something difficult as long as people are using the right tools for the job and the gaming industry need to wake up and smell platform diversity explosion that will soon be upon us. The only people I know that still argue against OS X on the desktop are the very same people that have never tried it. As for Linux, it tailors to the more computer literate and in my opinion it should stay that way. Still, the number of computer literate people should be enough to make it an interesting platform for entertainment providers.
nVidia gave a short OpenGL 3 presentation at SIGGRAPH 2007 which is available here.

08 May 08 The season’s afoot

Our media department was playing around in the parking lot outside the office and managed to take a couple of pictures of my car in the process. I have just taken it out of the garage for the summer and so far it seems to have survived the winter.

During the off season I replaced all the (original) pintle-type fuel injectors with disc-type injectors from Ford. The car was difficult to start when cold before but with the new injectors in place it fires up instantly.

I also did a lot of work remove rust from the undercarriage and rear axle, when I get some decent pictures of the end result I will post them together with the before-pictures. Hopefully before I get too much dirt under there.

There is still a lot of work to do but I’m well on my way.

28 Apr 08 axTLS, embedded security

Recently we were faced with the fact that our current encryption model isn’t strong enough to satisfy some of our more demanding customers. After discussing the possibilities of writing our own encryption model based on public strong encryption standards we finally decided to go for a SSL solution. Building upon an existing and proven standard carries much more weight than reinventing the wheel and I was faced with the task of locating a suitable SSL implementation for our embedded systems.

The first two implementations that popped into my head were of course GNU TLS and OpenSSL. After investigating both solutions I came to the following conclusions. GNU TLS has some external dependencies, such as libgcrypt and libgpg-error. As I want to keep dependencies to a minimum since we integrate them with our build system, which can be a painful process, that sort of spoke against GNU TLS. Though OpenSSL didn’t have any external dependencies we couldn’t already satisfy it had a problem which also plagued GNU TLS, it had a rather large footprint. Somewhere in the vicinity of five times larger than the binary it was going to be linked with. This was clearly something that would be difficult to motivate.

After some googling I discovered three very interesting alternatives, MatrixSSL, XySSL and axTLS. All designed for embedded systems and having an acceptable footprint.
MatrixSSL costs money for commercial use so I decided to push it to the back of the queue. XySSL had some weird problems on our platform resulting in the infamous “TANGO15 reboot“. axTLS, while only supporting TLSv1, worked flawlessly, was easy to compile and had a footprint of roughly 50% of the binary it would be linked with. Performance isn’t a factor for the application in questions and hence it was not investigated.

If you are looking for an SSL implementation for an embedded system and TLSv1 is good enough for you I can highly recommend axTLS.

23 Apr 08 Pigs do fly, Apple buys P.A. Semi!

According to this Forbes article Apple just bought P.A. Semi.

I was really sad when I found out that we weren’t going to see a PowerBook with a PWRficient chip. At the time I pretty much gave up on the idea of a powerful computer reaching home users anytime soon. I doubt we are going to see a PowerBook or PowerMac for that matter based on the PWRficient but at least they reignited the dream again.

21 Apr 08 Simple Buffer Parsing With C99

A friend of mine asked some questions relating to socket programming which spurred me to create a simple example based on an application I wrote a while back. It’s pretty basic stuff but I decided to post it here in case anyone has any feedback to offer.

I use flexible arrays where suitable so warm up your C99-compilers. There is virtually no error checking done in the example so be vary, also I didn’t compile it.

static const uint32_t PROTOCOL_MAGIC=0xdead001;
struct HelloHeader {
    uint32_t magic;
    uint32_t version;
};

enum Command {
    Command_NONE,
    Command_FILE_LIST,
    Command_GET_FILE,
};

struct CmdHeader {
    uint32_t cmd;
};

struct File {
    char name[256];
    uint64_t size;
};

struct FileList {
    uint32_t numFiles;
    File[] files;
};

struct GetFile {
    uint64_t len;
    /* null-terminated filename followed by data */
    char[256] name;
    char[] data;
};

uint64_t ntohll(uint64_t n)
{
#ifdef BIG_ENDIAN
    return n;
#else
    return (((uint64_t) ntohl(n)) << 32) + ntohl(n >> 32);
#endif
}

char* processCommand000301(char* buffer, size_t bufferLen)
{
    /* ... */
}

char* processCommand010000(char* buffer, size_t bufferLen)
{
    char* ptr = buffer;

    struct CmdHeader* commandHeader = (struct CmdHeader*) ptr;
    commandHeader->cmd = ntohl(commandHeader->cmd);
    ptr += sizeof(struct CmdHeader);
    switch (commandHeader->cmd) {
        case Command_FILE_LIST: {
            struct FileList* fileList = (struct FileList*) ptr;
            fileList->numFiles = ntohl(fileList->numFiles);
            ptr += sizeof(struct FileList);
            for (int i = 0; i < fileList.numFiles; ++i) {
                fileList.files[i].size = ntohll(fileList.files[i].size);
                printf("%s %lld\n", fileList.files[i].name, fileList.files[i].size);
                ptr += sizeof(struct File);
            }
            break;
        }

        case Command_GET_FILE: {
            struct GetFile* getFile = (struct GetFile*) ptr;
            getFile->len = ntohll(getFile->len);
            FILE* file = fopen(getFile.name, "w");
            fwrite(ptr, 1, getFile.len, file);
            ptr += getFile.len;
            fclose(file);
            break;
        }

        default:
            /* Unrecognized command! */
            return NULL;
    }

    return ptr;
}

int processBuffer(char* buffer, size_t bufferLen)
{
    char* ptr = buffer;

    struct HelloHeader* helloHeader = (struct HelloHeader*) buffer;
    helloHeader->magic = ntohl(helloHeader->magic);
    helloHeader->version = ntohl(helloHeader->version);
    if (helloHeader->magic != PROTOCOL_MAGIC) {
        /* Invalid header! */
        return -1;
    }
    ptr += sizeof(struct HelloHeader);

    while (ptr < (buffer + bufferLen)) {
        switch (helloHeader->version) {
            case 0x00010000:
                /* version 1.0.0 */
                ptr = processCommand010000(buffer, (size_t) ptr - buffer);
                break;

            case 0x00000301:
                /* version 0.3.1 */
                ptr = processCommand000301(buffer, (size_t) ptr - buffer);
                break;

            default:
                /* Unsupported version! */
                return -1;
        }
    }
    return 0;
}

P.S. I’m impressed by WordPress’ ability to consistently botch preformatted text when using the WYSIWYG editor. D.S.

07 Apr 08 Terrain rendering in Ada

A while ago I added another old project of mine to the projects page but I neglected to post about it.

The project is based on an old school project of mine from the university. It’s a terrain rendering engine written in Ada which was intended to be capable of handling very large terrains of very high detail, +/- 1 meter at the maximum zoom level if I recall correctly.

Since at the time the few OpenGL-bindings that existed for Ada were highly outdated and of varying quality I quickly decided I had to remedy the situation myself. It resulted in the AdaOpenGL project which to this very day, even though I haven’t updated it in over four years, still has a significant amount of downloads (according to SourceForge statistics). People have even tracked me down and called me personally about it, which was a first for me. For these very reasons I often have a guilty conscience for not bringing the project up to date. Two developers have offered to assist but neither of them every produced any releases.

After I had a viable OpenGL-binding I started on the terrain engine itself. As you might suspect I spent a lot of time on the binding and based on the little time I had left I decided to only implement a basic scene graph for the terrain engine. The idea was that someone else would pick up where I left off but as it turned out the company I did the project with never asked for my source code, which I found rather weird considering their initial enthusiasm.

I gave the project the name AdaTerrainEngine as I cannot recall what I initially named it. The data files belong to the company which I did the project with so I cannot release them. I don’t really remember how useful the source code is but at least it should provide a basic foundation for anyone who is doing OpenGL work in Ada.
The terrain engine is LGPL-licensed and AdaOpenGL is BSD-licensed.

P.S. I blatantly stole the design for the AdaOpenGL-site from an awesome CSS tutorial. After I finished the page I realized I forgot to credit the original author and I couldn’t find the site again. If you have any idea who did this please let me know! D.S.

14 Mar 08 Threadable C++ objects

I maintain a couple of objects at work which implement various behavior that requires a separate thread of execution. Since recently there were really only two of them but now they have multiplied beyond that so I realized that it would be appropriate to design a reusable framework in order to reduce unnecessary code duplication. The design I came up with is based around an interface which defines the API of any “threadable” object and an abstract implementation which provides the means to synchronize the threads. Threads in our systems tend to have a very long life span and we avoid any kind of resource allocation (including launching threads) which are soon thereafter released as that can lead to fragmentation.

Although I’m fairly satisfied with this implementation there is at least one flaw in it. I’ll leave it to my readers to find it.

The interface is fairly simple and provides three methods. kill() which is used to signal thread termination, running() which is used to query whether the thread is actually up and running and finally the actual thread function implemented as operator()().

class IThreadable {
public:
    virtual ~IThreadable() throw() { }

    virtual void kill() = 0;
    virtual bool running() const = 0;
    virtual void operator()() throw() = 0;
};

The AbstractThreadable object implements the kill() and running() methods which should look the same across all kinds of threadable objects. It also implements two protected methods which must be called in the threadable object to signal that it has enter or exited its thread function, these are known as signalThreadRunning() and signalThreadExit().

I’m using a tristate variable to keep track of the threads current state of execution. The reason behind this is that I want to differentiate between a thread that has not yet been started. If someone calls kill() and never actually launches the thread the kill() method will block on the semaphore unless we can detect this state.

enum ThreadState {
    ThreadState_NONE,
    ThreadState_RUNNING,
    ThreadState_KILLED,
};

class AbstractThreadable : public IThreadable {
public:
    AbstractThreadable() : m_state(ThreadState_NONE) { }

    virtual void kill() {
        scoped_lock stateLock(m_stateMutex);
        if (m_state != ThreadState_KILLED) {
            ThreadState prevState(m_state);
            m_state = ThreadState_KILLED;
            stateLock.unlock();
            if (prevState == ThreadState_RUNNING) {
                m_threadExitSemaphore.p();
            }
        }
    }

    virtual bool running() const {
        if (scoped_lock(m_stateMutex), m_state != ThreadState_KILLED) {
            return true;
        }
        return false;
    }

    virtual void operator()() throw() = 0;

protected:
    virtual void signalThreadEnter() {
        scoped_lock stateLock(m_stateMutex);
        m_state = ThreadState_RUNNING;
    }

    virtual void signalThreadExit() {
        m_threadExitSemaphore.v();
    }

private:
    ThreadState m_state
    mutable mutex m_stateMutex;
    semaphore m_threadExitSemaphore;
};

Finally I will present a simple object using the above implementation in order to implement a threadable object which just sleeps until someone signals it to terminate via kill(). Notice how signalThreadRunning() and signalThreadExit() are used in order to notify the AbstractThreadable implementation of the current thread state.
Note: Calling running() requires locking a mutex so if the thread loop is very tight it might be wise to unroll it.

class Sleeper : public AbstractThreadable {
public:
    void operator()() throw() {
        signalThreadEnter();
        while (running()) {
            sleep(1);
        }
        signalThreadExit();
    }
};

This is a simple example showing how to use the sleeper thread. I’m using Boost to create the thread. thread is a thread object and ref makes sure the sleeper object is passed by ref rather than being copied.

int main(int argc, char **argv)
{
    Sleeper sleeper;
    thread sleeperThread(ref(sleeper));
    ...
    sleeper.kill();
    return sleeperThread.join();
}

29 Feb 08 sed stupidity or ampersand magic?

We were reformatting some code at work the other day using find and sed and stumbled upon a very weird case when using the ampersand (&) character. We accidentally expanded all boolean and expressions in our C++-code (&&) to “& &” when applying a rule to modify “<type> & <variable>” to “<type>& <variable>”. In order to change them back we ran a “find . -type f | xargs sed -i -e ’s:& &:&&:g’” which resulted in all the expression becoming “& && &”?!

After some fiddling around it turns out the ampersands in the resulting string had to be escaped to reach the desired result. As far as I know ampersands are not part of the regular expression syntax and I’ve never before had to escape anything in the resulting string.

In the end the following syntax yielded the correct result.

echo " & & " | sed -e 's:& &:\&\&:g'

 &&

I don’t know if this is a bug or feature and if anyone else does please let me know.