ChaosEngineer.com

Ooops, I definitely got sidetracked more mucking around with my platform independent rendering engine. Instead of just holding out on the NDS wifi article in silence for any longer, I thought I would chime in and let you know I haven't forgot about you and show you what I've been up to.

My platform independent rendering engine is based on a pure virtual renderer base class that is filled with an instance of a derived class by a dynamically linked code module. Along with the pure virtual renderer class, there are a few pure virtual resource classes as well, the actual instantiation of which are performed by the renderer class. Trying to maintain platform independence, the singleton (factory / manager) class that returns this renderer instance has some interesting code I thought I would share.

It is possible to have a project directory (even though a bit messy) that will compile in both Microsoft Visual Studio and in GCC without having to change a lick of code. Thats child's play for a simple project, but its still possible even for a project that contains a main executable and several other dynamic libraries.

So first off, since we are dynamically linking, we are definitely going to have function pointers returned by dlsym(...) or GetProcAddress(...). The actual DLL/SO modules export only three functions--one to create the instance, one to destroy it, and one to return the version. So we have the typedefs for these as so:

 
typedef int (*fpCreateRendererInterface)(CRenderer **pInterface);
typedef int (*fpDestroyRendererInterface)(CRenderer **pInterface);
typedef int (*fpQueryRendererVersion)(void);
 

Second off, this is platform independent, so obviously we will need to be using some fancy pre-processor conditionals (PPCs) to detect the platform / compiler. Now in certain instances we only want to know what the compiler is, because the actual platform doesn't matter, but in other cases we want to know the actual platform as well because the same compiler is used on multiple platforms (GCC is used in Linux and also Mac). First for the compiler, we can detect Visual C++ by checking to see if _MSC_VER is defined, and we can detect GCC by checking to see if __GNUC__ is defined. For the platform, we can detect Windows by checking if WIN32 is defined (yes, even 64-bit Windows defines this), we can detect Linux by checking if __linux__ is defined, and we can detect OSX by checking if __MACH__ and __APPLE__ are defined.

We can then use these PPCs to create code that is savagely flexible. In this singleton class, I have a static void pointer to the library (Windows' HMODULE is just a void*, don't let them trick you) called m_hLibrary. Lets see how we could dynamically link either a .so file in Linux or a .dll file in Windows and place the returned handle in the same variable... here is the code for the "SGL" (SDL GL) renderer type which can be used in either Linux or Windows, taking into account if the code is in release or debug mode:

 
case rstSGL:
	Log::Entry(-1,__FILE__,__LINE__,"Creating SGL rendering interface...");
#if defined(__GNUC__)
 
#if defined(NDEBUG)
	Log::Entry(-1,__FILE__,__LINE__,"__GNUC__ defined... opening release library");
	m_hLibrary = dlopen("./libRendererSGL.so",RTLD_NOW);
#else
	Log::Entry(-1,__FILE__,__LINE__,"__GNUC__ defined... opening debug library");
	m_hLibrary = dlopen("../RendererSGL/bin/Debug/libRendererSGL.so",RTLD_NOW);
#endif
	if(!m_hLibrary)
		Log::Entry(2,__FILE__,__LINE__,"Error loading shared library. dlerror() = %s",dlerror());
#elif defined(_MSC_VER)
#if defined(NDEBUG)
	m_hLibrary = LoadLibraryExA("RendererSGL.dll",NULL,NULL);
#else
	m_hLibrary = LoadLibraryExA("../RendererSGL/bin/Debug/RendererSGL.dll",NULL,NULL);
#endif
#endif
	break;
 

Please note that NDEBUG is not defined by GCC even when using -O2, so you need to set your release target to define this manually. I find it strange because it is supposedly a standard...

Also note that when linking using dlopen(...), we specify RTLD_NOW to force resolution of all symbols in the dynamic library. If we used RTLD_LAZY, we could exclude a lot of code from our library, and let it resolve symbols later (i.e. from the main executable). Unfortunately this works great on Linux but seemingly has no analogy in Windows, and in order to keep the projects symmetrical across platforms, we use RTLD_NOW to behave appropriately.

So lets look at whats going on here... its a bit over complicated, but I thought I would provide some extra good ideas for you guys. If using GCC, we use dlopen(...) to get the handle to the .so file, and if using VC++, we use LoadLibraryExA(...) to get the handle to the .dll file. If NDEBUG is defined, we link to the release version of the dynamic library (actually look in the current directory or system path as if it were a proper deployment), otherwise we link to the debug version of the library to simplify development.

Oh snap! So we now have linked to a dynamic library regardless of what platform we are running on! This is kickass, so where to now? We need to get pointers to the functions in the library we are going to use. The function that does this in Linux is dlsym(...), and in windows is GetProcAddress(...). Here is what this would look like in a platform independent form:

 
#if defined(__GNUC__)
	fpCreateRendererInterface IntCreate=(fpCreateRendererInterface)dlsym(m_hLibrary,"CreateInterface");
#elif defined(_MSC_VER)
	fpCreateRendererInterface IntCreate=(fpCreateRendererInterface)GetProcAddress((HMODULE)m_hLibrary,"CreateInterface");
#endif
 

Damn, its that easy? Indeed it is. We now have a function pointer to a procedure in a dynamically linked library regardless of if it came from a .so file in Linux or a .dll file in Windows. Obviously once armed with the function pointer, the remaining code is the same regardless of platform. We just use IntCreate(...) like it were a normal function. In this case we pass a pointer to a pointer to the pure virtual renderer base class, and inside the dynamic library, we assign the pointer to an instance of a derived class:

 
extern "C"
{
	int CreateInterface(CRenderer **pInterface)
	{
		if(*pInterface)
			return -1;
 
		*pInterface = new CRendererSGL();
 
		return 0;
	}
...
 

This is really quite powerful, and since the only actual call to an exported function is when creating the derived class instance, the performance hit while using the derived class is only from the vftable. More importantly this lets us do something ultra simple in our main code to get a reference to an abstract renderer interface that doesn't require knowledge of the nitty-gritty of either D3D or OpenGL:

 
	Log::Entry(0,__FILE__,__LINE__,"Creating rendering device...");
	Graphics::CreateInterface();
	CRenderer *renderer = Graphics::GetInterface();
	renderer->Initialize(0);
 
	if(g_bSafeDevice)
		renderer->CreateDeviceSafe();
	else
		renderer->CreateDevice(800, 600, 32, g_bWindowed);
 

Yeah, I know. That code is delicious. Its even more tasty when you have a CMesh that contains instances of pure virtual CResourceVtxBuff and CResourceIdxBuff created by the derived CRenderer class in the dynamic library, and CModel that contains a number of CMesh instances as well as a number of CMaterial instances that contain instances of pure virtual CResourceTexture also created by the derived CRenderer class in the dynamic library. So to create a model and render it, you would have to do something like the following:

 
	// example of loading a mesh
	mshOut = new CMesh();
	iStride=mshOut->GetVtxStride();
	mshOut->SetVtxCount(iVertexCount);
	mshOut->SetFaceCount(iFaceCount);
 
	m_pRenderer->CreateMeshBuffers(iStride*iVertexCount,sizeof(int)*3*iFaceCount,mshOut);
 
	// if pRenderer is D3D, this lock would be like IDirect3DVertexBuffer9->Lock(...),
	//while in OpenGL it would be like glMapBufferARB(...). Transparent at this point.
	VtxData=(unsigned char*)mshOut->GetVtxPtr()->LockWrite(0,0);
 
	// Write vertex buffer to VtxData here...
 
	mshOut->GetVtxPtr()->Unlock();
 
	IdxData=(unsigned char*)mshOut->GetIdxPtr()->LockWrite(0,0);
 
	// write index buffer to IdxData here..
 
	mshOut->GetIdxPtr()->Unlock();
 
	// example of loading a material
	mtlOut=new CMaterial();
	mtlOut->SetDiffuse(FloatToLongColor(v3fDiffuse.x, v3fDiffuse.y, v3fDiffuse.z));
	mtlOut->SetAmbient(FloatToLongColor(v3fAmbient.x, v3fAmbient.y, v3fAmbient.z));
	mtlOut->SetSpecular(FloatToLongColor(v3fSpecular.x, v3fSpecular.y, v3fSpecular.z));
	mtlOut->SetMaterialName(sMaterialName);
 
	mtlOut->SetTexture(m_pRenderer->CreateTexture(&imgTexture));
 
	// and you add them to a model which contains std::vectors of meshes and materials...
 
	iMaterialIdxList[j] = mdlOut->AddMaterial(mtlOut);
 
	//...
 
	mdlOut->AddMesh(mshOut,iMaterialIdxList[iMaterialRef]);
 
	// and eventually render!
 
	iMeshCount = mdlCurr->GetMeshCount();
 
	for(j=0;j<iMeshCount;j++)
	{
		pRenderer->SetWorld(&matWorld);
 
		k = mdlCurr->GetMeshMaterialIdx(j);
 
		pRenderer->SetTexture(mdlCurr->GetMaterial(k)->GetTexture());
		pRenderer->Render(mdlCurr->GetMesh(j));
	}
 

Anyway, I guess this isn't all that useful, and maybe I'm just showing off at this point. It does exhibit some good ideas, and show how glorious abstraction can be though. Regardless, I got distracted again. The NDS Wifi example continues. I will post in the next few days to describe the protocol used and give a little primer on sockets. Keep your eyes peeled.

So the idea I had to display the potential for socket programming on the NDS was to simply pipe the input data from the NDS over the network to a remote host. To make this appealing from beyond just a technical standpoint, I figured the remote host should be running some sort of visualization that the user could interact with remotely by using the NDS touchscreen. Making an OpenGL application with a 3D cube the user could spin using the touchscreen would work nicely. Lets figure out how to do that from an OpenGL perspective.

We want to rotate the cube along an axis perpendicular to the mouse motion vector. This will give the impression that the cube is spinning as a direct result of the motion, as if you had quickly ran your hand along it. So we first need to construct a vector that is perpendicular to the motion, and lies in the plane of the screen. Lets assume we have a default OpenGL view setup, where we are looking along the negative z-axis, and the screen lies in the xy-plane. Thus, we want a vector that is perpendicular to the mouse motion and lies in the xy-plane.

Anyone with some math background may immediately recognize the cross-product as a candidate to construct the perpendicular vector. Indeed, if we create a vector from the relative mouse motion (conveniently in our environment the mouse motion is on the xy-plane, so the screen coordinates align nicely), then take the cross-product of this vector with the z-axis, the result is a vector that is perpendicular to the mouse motion and lies in the xy-plane.

 
 
//iRelX and iRelY is the relative mouse motion
 
float fTheta;
float vCross[3];
float fCrossMag;
 
// Assign theta to the magnitude of the mouse motion vector
// This is rather arbitrary
fTheta = (iRelX*iRelX + iRelY*iRelY)/16.0;
 
/*
vCrossX = v1y*v2z - v1z*v2y
vCrossY = v1z*v2x - v1x*v2z
vCrossZ = v1x*v2y - v1y*v2x
*/
 
// Perform cross-product.
// Here v1 is the mouse motion vector and v2 is the z-axis.
// This gives us a vector that is perpendicular to
//the mouse motion and lies in the xy-plane.
vCross[0] = iRelY*1.0 - 0.0*0.0;
vCross[1] = 0.0*0.0 - -iRelX*1.0;
vCross[2] = -iRelX*0.0 - iRelY*0.0;
 
// Normalize this vector for use as axis in rotation matrix
fCrossMag = sqrt(vCross[0]*vCross[0] + vCross[1]*vCross[1] + vCross[2]*vCross[2]);
vCross[0] = vCross[0] / fCrossMag;
vCross[1] = vCross[1] / fCrossMag;
vCross[2] = vCross[2] / fCrossMag;
 

Great. So now we can just take this vector and use it when calling glRotatef(...) in order to transform the modelview matrix, right? Well, not quite. We CAN do this, but if you've played around with glRotatef(...) to manipulate objects in your scene via the modelview matrix, you may have noticed that when you rotate an object on one axis, it actually transforms the other two axes. This basically means that using glRotatef(...) or any other matrix transform in this manner is actually operating in the model's local coordinate system. If we tried to rotate our cube around our constructed axis using glRotatef(...), it would work as expected initially but then subsequent rotations would rotate around the already transformed axis. We don't want this, we want to operate on a fixed coordinate system, what is often referred to as the world coordinate system.

This issue arises because matrix operations post-multiply on the matrix stack in OpenGL. If we could pre-multiply the operations, we would be good, but unfortunately there is no way to tell OpenGL to pre-multiply instead, so we really have to do this operation manually. In order to accomplish this, we need to maintain our own modelview matrix accumulation, and each frame multiply our rotation matrix by this accumulation matrix to get the new modelview transform. Note that if we used glRotatef(...), it would multiply the accumulation matrix by the rotation matrix. Matrix multiplication isn't commutative, and in our situation it makes all the difference in the world.

So we need to get our hands dirty with a little linear algebra. First we need to define a few 4x4 matrices that we can hand to OpenGL:

 
// OpenGL likes this format for a 4x4 matrix
//as declared in the Redbook
float matWorld[16];
float matWorldAccum[16];
float matBuff[16];
 

Also define a function that can construct a rotation matrix when rotating around an arbitrary axis:

 
// Theta is in radians
void MatrixRotation(float *matRot, float fTheta, float fAxisX, float fAxisY, float fAxisZ)
{
	float fCosTheta, fSinTheta;
 
	// m0   m4   m8   m12
	// m1   m5   m9   m13
	// m2   m6   m10  m14
	// m3   m7   m11  m15
	fCosTheta = cos(fTheta);
	fSinTheta = sin(fTheta);
	matRot[0] = fAxisX*fAxisX+(1.0-fAxisX*fAxisX)*fCosTheta;
	matRot[4] = fAxisX*fAxisY*(1.0-fCosTheta)-fAxisZ*fSinTheta;
	matRot[8] = fAxisX*fAxisZ*(1.0-fCosTheta)+fAxisY*fSinTheta;
	matRot[12] = 0.0; // m12,m13,m14 can be used for translation
	matRot[1] = fAxisX*fAxisY*(1.0-fCosTheta)+fAxisZ*fSinTheta;
	matRot[5] = fAxisY*fAxisY+(1.0-fAxisY*fAxisY)*fCosTheta;
	matRot[9] = fAxisY*fAxisZ*(1.0-fCosTheta)-fAxisX*fSinTheta;
	matRot[13] = 0.0;
	matRot[2] = fAxisX*fAxisZ*(1.0-fCosTheta)-fAxisY*fSinTheta;
	matRot[6] = fAxisY*fAxisZ*(1.0-fCosTheta)+fAxisX*fSinTheta;
	matRot[10] = fAxisZ*fAxisZ+(1.0-fAxisZ*fAxisZ)*fCosTheta;
	matRot[14] = 0.0;
	matRot[3] = 0.0;
	matRot[7] = 0.0;
	matRot[11] = 0.0;
	matRot[15] = 1.0;
 
}
 

Now we need to be able to multiply our rotation matrix by the accumulated modelview transformation matrix:

 
void MatrixMultiply(float *matDest, const float *matA, const float *matB)
{
	// m0   m4   m8   m12
	// m1   m5   m9   m13
	// m2   m6   m10  m14
	// m3   m7   m11  m15
 
	// Inner product of each row with each column
	matDest[0]=matA[0]*matB[0] + matA[4]*matB[1] + matA[8]*matB[2] + matA[12]*matB[3];
	matDest[4]=matA[0]*matB[4] + matA[4]*matB[5] + matA[8]*matB[6] + matA[12]*matB[7];
	matDest[8]=matA[0]*matB[8] + matA[4]*matB[9] + matA[8]*matB[10] + matA[12]*matB[11];
	matDest[12]=matA[0]*matB[12] + matA[4]*matB[13] + matA[8]*matB[14] + matA[12]*matB[15];
 
	matDest[1]=matA[1]*matB[0] + matA[5]*matB[1] + matA[9]*matB[2] + matA[13]*matB[3];
	matDest[5]=matA[1]*matB[4] + matA[5]*matB[5] + matA[9]*matB[6] + matA[13]*matB[7];
	matDest[9]=matA[1]*matB[8] + matA[5]*matB[9] + matA[9]*matB[10] + matA[13]*matB[11];
	matDest[13]=matA[1]*matB[12] + matA[5]*matB[13] + matA[9]*matB[14] + matA[13]*matB[15];
 
	matDest[2]=matA[2]*matB[0] + matA[6]*matB[1] + matA[10]*matB[2] + matA[14]*matB[3];
	matDest[6]=matA[2]*matB[4] + matA[6]*matB[5] + matA[10]*matB[6] + matA[14]*matB[7];
	matDest[10]=matA[2]*matB[8] + matA[6]*matB[9] + matA[10]*matB[10] + matA[14]*matB[11];
	matDest[14]=matA[2]*matB[12] + matA[6]*matB[13] + matA[10]*matB[14] + matA[14]*matB[15];
 
	matDest[3]=matA[3]*matB[0] + matA[7]*matB[1] + matA[11]*matB[2] + matA[15]*matB[3];
	matDest[7]=matA[3]*matB[4] + matA[7]*matB[5] + matA[11]*matB[6] + matA[15]*matB[7];
	matDest[11]=matA[3]*matB[8] + matA[7]*matB[9] + matA[11]*matB[10] + matA[15]*matB[11];
	matDest[15]=matA[3]*matB[12] + matA[7]*matB[13] + matA[11]*matB[14] + matA[15]*matB[15];
 
}
 

Great, so we are now equipped to construct and maintain our own modelview matrix! This means we no longer have to ask OpenGL for transformations. We can do them ourselves, and just hand the transformation matrix to OpenGL for rendering the current scene.

 
// Now we contruct a matrix to rotate around this axis
MatrixRotation(matWorld, fTheta*PI/180.0, vCross[0], vCross[1], vCross[2]);
 
// Notice this is matWorld * matWorldAccum, instead of
//matWorldAccum * matWorld, as it would be if we just used
//glMultMatrix with the constructed matWorld rotation matrix
MatrixMultiply(matBuff, matWorld, matWorldAccum);
 
// Assign the world matrix accumulation to the modelview matrix
glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(matBuff);
 
// Update accumulated matrix
for(i=0; i<16; i++)
	matWorldAccum[i] = matBuff[i];
 

Fun stuff, eh? I always feel empowered by doing more things in my client code instead of processing data in a black box. Next part of the article we will get into the NDS and socket programming. I know you are holding your breath. =)

I recently got distracted and started porting my old game engine to use a new platform independent rendering interface I devised. I'm pretty psyched about the rendering system, it fully encapsulates a graphics sdk and all it's associated resources (vertex and index buffers, textures, etc). Deriving from the pure virtual renderer and resource base classes, I can write a new renderer class that targets a specific platform/SDK, plug it in, and just have it work without modifying a single line of code.

Part of this old game engine was a resource management class that could load 3D Studio MAX ASCII exported scenes/objects. These files are .ASE files, which is a versatile and simple format that has a surprisingly wide usage by various game engines. ASE files can be used when building Doom3 and Quake3 / Quake4 levels, since the format boasts native support in ID and Loki software's GtkRadiant design tool, so it is a generally accepted format for storing game content. The ASE format specification itself is quite versatile, and can contain material specifications, verticies, faces (vertex indexes), vertex colors, texture coordinates, and even animations with full interpolation parameters just to name a few. Being ASCII, it is rather easy to parse and modify.

In the ASE format, each line of the file serves a single purpose. It is either an opening or closing tag for a field, or an element in a field. It made sense then when I first wrote the import code to parse the file on a line-by-line basis using ifstream::getline(...). This worked great in Windows, and I had no issues with my implementation. Flash forward to now when I am rewriting this code to work in Linux, but using files created in Windows. I never thought the disparities of a 'newline' between platforms would ever cause such a problem.

The ifstream::getline function was specifically designed to get a single line from a file without including the newline or delimiting character(s). The newline character ('\n') is the delimiting character by default, as it serves its purpose most of the time. ifstream::getline(...) reads in all characters up to the delimiter, then extracts and discards the delimiter. The problem arises when considering the definition of a newline on different platforms.

In Windows, the newline for iostream operations is a carriage-return,line-feed combo. It is actually two characters with ASCII decimal values 13, 10 (0x0D, 0xOA in hex), often referred to as a CRLF combo. In Linux, a newline is simply a line-feed (dec 10, hex 0x0A). So when performing ifstream::getline(...) in Linux, what you are doing is reading a line up to the delimiter--newline by default--and if your file was created in Windows only the last half of the CRLF combo is discarded, while the first half is read in as the last character in your string variable. This makes any literal comparisons made against this string variable fail in Linux while working fine in Windows.

I thought for sure there was some elegant way to overcome this and attain platform independent file parsing without forcing a complete rewrite of the import code. This turned out to not be the case. The ifstream::getline function is handy, but by doing you the favor of automatically extracting and discarding the delimiter, it somewhat cripples it's versatility. My impetus was two fold. Obviously first I wanted to be able to read these ASE files with 0x0D,0x0A line delimiters into Linux without having the 0x0D on the end of my read lines. This would be simple enough, but I wanted also code that would work regardless of platform combinations. This meant reading Windows files in Linux, Linux files in Windows, Windows files in Windows and Linux files in Linux.

So this added a bit of complexity to the requirements. First off, it would be impossible to use ifstream::getline(...) because it would extract and discard different characters from the file stream depending on platform. When reading a Linux file in Windows, it would keep reading from the stream, looking for a CRLF combo, and would likely read the entire file without finding one. When reading a Windows file in Linux as I found out, it would read the line including 0x0D, then stop at and discard the 0x0A. If it didn't consider reading Linux files in Windows, the following implementation was elegant, and worked in all other situations:

 
char szLineBuff[256];
ifstream fsIn("test.ase", ios::in);
 
// Note that failbit would be set if this
//failed to extract any chracters before
//the delimiter, and the loop would end.
while(fsIn.get(szLineBuff, 128, '\n').good())
{
	// This extracts 2 characters in
	//windows, just 1 in Linux.
	fsIn.ignore(2,'\x0A');
	// Cleans Windows line in Linux
	if(szLineBuff[strlen(szLineBuff)-1] == '\x0D')
		szLineBuff[strlen(szLineBuff)-1] = '\0';
 
	// Process szLineBuff...
 
}
 

Now I recognize that opening the file in binary mode (ios::binary) would make the '\n' be interpreted as simply '\x0A' on both Windows and Linux. My problem with this is that the file isn't binary. I'm stubborn and think this ASCII text file should be read in text mode, and that there should be a simple way to interpret newlines from any platform on any other platform. There must be a simple and elegant solution out there somewhere...

I'll be honest and say I stopped there, as my code worked for 3/4 of the scenarios I laid out. I have yet to find a Linux native source of ASE files, so the last scenario (Linux files on Windows) hasn't become important for me yet. If anyone has a better solution for this, or knows one that would work in all scenarios while still being somewhat elegant, I'd love to hear about it.

Also, I know I promised a follow-up on the NDS WiFi programming. I only got a little distracted, the example applications are nearly finished. You'll definitely like it when it comes.

I have a grand plan to make a beowulf cluster dedicated to computing, interfacing and manipulating chaotic systems. I have a protocol devised that would allow multiple applications running on an arbitrary number of computers in a network pool hardware resources for this task. Each application (node) would have it's own capabilities, and could be used to collect sensory data, provide a human interface, or just to crunch numbers. I could easily have a management node that would maintain a list of other nodes and their abilities, and query them as needed. I recently had a great idea to use a NintendoDS as both an interface (via the touchscreen), and a sensory device (collecting audio samples with the buit-in microphone).

So I did a bunch of NDS programming a few years back using devkitPro. They were at release r20 then, and dswifi had just made it on the scene. For those who don't know, dswifi is a library written by Stephen Stair that allows NintendoDS homebrew developers using devkitPro to use the wifi features of the NDS. You can connect to wireless access points, and from there you have network access and a standard Berkley sockets interface, so you can pretty much accomplish anything you set your mind to. I just downloaded the latest version of devkitPro, which is now r26. I was quite pleased with the progress the toolchain had made during my absence, and especially pleased to see dswifi so well integrated. Things pretty much went downhill from there.

NOTE: 09.06.16 - The rest of this post is for historical information only. After I isolated the bug in dswifi that was the root of my problems, I notified the maintainers of devkitPro who committed a fix. There are now new releases of libnds, dswifi and default arm 7 available that correctly handle the DS' MAC address. Please see the addendum at the end of this article for more information. Do not downgrade your devkitPro install if you are having problems. Instead get the very latest versions of all libraries, and you should be good.

There were no examples provided with the nds-examples package that showed how to connect to WEP secured Access Point (AP). The only examples provided were for either using the WFC information stored in the NDS firmware, or for connecting to an unsecured AP. They did have code to enumerate APs and to provide a keyboard interface that I happily integrated. Prior to even testing the included examples I poured a couple hours into writing a code base using the most involved methods possible (swapping out all the simple 0 argument init functions with the 8 argument equivalents) to establish a console and collect and display a list of APs. Selecting an AP from this list that was secured would then provide a prompt to specify either 40-bit or 104-bit encryption. The user could then type in the 10 or 26 character string using a tiny little keyboard on the DS touchscreen. This string was then encoded to a byte array and passed to the Wifi_ConnectAP(...) function provided by dswifi.

I then spent three times as long troubleshooting. It took a while for me to remember that I have MAC address filtering enabled on my router (*bonk self*). I loaded up a wifi enabled game, pulled the MAC address of my DS from Nintendo's WFC interface, and put it on my router's Access Control List. I rubbed my hands in greedy anticipation, and loaded up my homebrew app once again. No joy ensued. I still could not connect to my AP.

At this point, I decided to try something simple just to verify the functionality of dswifi (yes, I'm backasswards when it comes to programming), so I loaded up one of the basic dswifi examples. Strangely, it did not work either. I knew it wasn't my DS or router anymore, because I would connect with a proper DS game that was wifi enabled. I didn't see anyone else complaining in the devkitPro forums, but r26 was *just* released and the forums over there aren't exactly teeming with DS programmers. I thought maybe the devkitARM toolchain for x64 Linux might have been the culprit, so I tried in Windows XP as well. Still no joy.

Operating under the assumption that dswifi worked at SOME point in devkitPro, I decided it was time to start digging backwards through the releases, and testing the functionality as I went with the most simple example possible... one that loaded the AP information from the WFC info stored in the DS and automatically tried to connect. This simple example did indeed work back in devkitPro r25 (2009.02.20). I then increased my resolution to try and find *exactly* where dswifi broke in devkitPro.

<SNIP>
Several paragraphs removed at the request of the devkitPro maintainers.
This section basically highlighted my procedures to somewhat subvert the available devkitPro library distributions to create a tool chain that pre-dated issues with the DS MAC address handling. As noted above, the bug I isolated has been fixed in the latest devkitPro library distributions, so this information is no longer pertinent. I found out that in my setup, the simple WFC autoconnect example worked fine when built with dswifi 0.3.7, but refused to work in dswifi 0.3.8. So I configured a motley devkitPro toolchain with different versions of specific libraries in order to get dswifi working for me.
</SNIP>

I now have code running on my DS that allows me to connect to my WEP secured AP, and from there get a socket connection to my workstation. Its pretty blasted awesome to be able to telnet into your DS and see what you type into your computer appear on the screen. Its several magnitudes more awesome to use the DS touchscreen and keys to remotely manipulate a 3D chaotic system in realtime. Add the ability to pipe recorded audio data from the DS microphone over the network and into that system where the fourier-tranformed signal is then used to modulate said system, and you might as well be dividing by zero.

UPDATE 2009.06.07:

After butting heads with the maintainers of devkitPro on the forum (they deleted a few of my posts because they don't like people telling users to downgrade or linking to obsolete libraries -- support headaches) who said the examples worked fine in r26, I pulled the source code for dswifi 0.3.7 and 0.3.8 than ran diff against the versions, excluding the svn nonsense:

diff -urNp -x '*svn*'  dswifi-src-0.3.7 dswifi-src-0.3.8 > dswifi.patch

Thanks to diff and a little persistence reading through the patch file, I found exactly where my problem was originating.

Remember how I said I had MAC address filtering enabled on my router? Well recent changes to dswifi did indeed break the example, for me at least... the MAC address bytes were being read from the firmware but written to a main data structure incorrectly:

 
for(i=0;i<3;i++)
	WifiData->MacAddr[i]= ReadFlashByte(0x36+i) + (ReadFlashByte(0x36+i+1)<<8);
 

This loop doesn't write the data to MacAddr as intended. After the loop, MacAddr is filled with the bytes from the following firmware addresses : 0x36, 0x37, 0x37, 0x38, 0x38, 0x39. The intention is obviously to write the bytes from 0x36, 0x37, 0x38, 0x39, 0x40, 0x41. Simple mistake, simple solution. I notified the maintainers, so when you download devkitPro r27, you can thank me when connecting to APs with MAC address filtering. =)

ADDENDUM 2009.06.16:

A fix for the MAC address bug in dswifi was submitted to the devkitPro svn on June 11th. The maintainers built new distributions of three libraries the same day. So if you get the following versions of libnds, dswifi and default arm7, your DS' MAC will be properly presented to access points:

libnds 1.3.5
dswifi 0.3.9
default arm7 0.5.4

Way to go, devkitPro.

You can see the full discourse on the forums here.

With this issue now past tense, I'll work on posting a PART 2 of this article with some actually useful programming information.

Over the past couple years I have completely migrated out of Microsoft Windows except for those circumstances where I am forced to use it (read: 'work'). I have thus been spending my time discovering the amazing power of Linux and of the opensource community. OpenGL has proved to be quite a good friend. I am constantly enheartened by it's prowess. VBOs, PBOs, FBOs... OpenGL is far from a substandard graphics API (as I used to believe) and can push polygons with the best of them. SDL has also been by my side, cutting a path through the uglier bits of interfacing with X11, providing clean input methods, and making my projects easily run on both Windows and Linux.

I used to be a Microsoft fan boy. I loved DirectX, and spent nearly every waking hour using it while crafting a game engine. I can remember working with Direct3D 6 and tearing my hair out trying to decide between immediate and retained mode. What an ugly API it was too, but I still used it over OpenGL because I had the feeling it was more powerful. DirectX 8 came out and all of a sudden everything was sleek and seemed to fit into place so well, and boy, was I ever excited over DirectX 9.

So what happened? At what point did I decide that Microsoft was evil and didn't deserve my time? The feeling culminated slowly. Microsoft pulled one stunt after another, betraying the trust consumers had placed in them. They were constantly using their market share and financial prowess to inhibit innovation and force the consumer's hand to benefit only themselves. It soon became apparent to me that they knew almost no bounds. They would commit horrible atrocities and get away with it because they are Microsoft.

I had been following the development of Bungee's Halo eagerly for some time. This was going to best PC game ever. One day, the Halo website just disappeared. Nobody knew what happened. Microsoft had recently announced they were entering the console market. It soon became obvious what was going on. Halo was to be a launch title for Microsoft's Xbox. They had acquired Bungee. I held my breath and patiently waited for the PC release of Halo. Bungee wouldn't turn it's back on the PC market, would they?

When it finally came, the PC port of Halo was substandard. I played the demo and never got the full version, maybe because Blood Gulch was enough for me, or maybe it was the flaws I saw. The networking code was seemingly never designed to operate with the high latencies found on the WAN. It was possible to accomplish... Unreal Tournament's cubic interpolation allowed the server to accurately render players and detect hits based on their projected locations. When playing the PC port of Halo, I found myself having to perform this interpolation myself, aiming ahead of players a certain distance based on the amount of latency I had with the server. Don't get me wrong, I got damn good at this and could still drop mofos from the far side of Blood Gulch with a pistol while both at full tilt, but I still felt betrayed and forgotten by Bungee with their new guiding hand. And man, oh man, when Halo 2 was released on the PC under Microsoft's new "Games for Windows" (AKA: Games that _only_ run on MS's craptacular new Vista OS) platform, there aren't words to describe how furious I was. Bungee's output was again used to push Microsoft's own agenda, this time trying to wrangle people into buying the new Vista OS, which was almost universally hated. Halo and Bungee aside, there are plenty of other reasons to despise Microsoft...

I had always been a tables layout HTML kind of guy. Who wasn't? When CSS starting making its way into the mainstream and new standards for positioning web page elements were devised, I resisted change for quite some time. Tables were so handy. I didn't need these new fancy DIVs and all this new positioning mojumbo. I could do all that with nested tables using colspan, rowspan, and the ever-useful CENTER tag. Needless to say, the CSS revolution didn't move me. I was still happily making webpages without a single style attribute. I tell you this just so you know I'm not a standards whore who expounds upon and proudly touts every little tidbit thrown out by the W3C.

One day while meandering about on the interwebs and mourning the fact VRML never caught on, I stumbled across a new standard that made me salivate. Scalable Vector Graphics had become a web standard. The W3 Consortium said so! This was blasted glorious! It was only a matter of time before webpages based on SVG would become standard, and we could present content on interactive SMIL animated geometry! SVG had presented epic new parameters to the web.

I quickly downloaded Opera with it's high level of SVG and SMIL compliance, and set to work creating an exemplary SVG based webpage. Previous experience with geometry and transformation pipelines from game engine design lent itself well here. I learned all sorts of new things with Javascript, stuff I had never before even thought of doing. I was dynamically modifying SVG transform attributes, adding and removing JS event handlers, and generally having a ball. I made a tasty little SVG "module" that would allow you to specify an array of images and it would create a task bar much akin to the Mac OS X dock. As your mouse cursor approached one of the "tasks" the image associated with it would grow larger( transform="scale(4.3*glory, 5.2*splendor)" ), nudging it's neighboring task icons to either side. You could even grab the dock itself and drag it to some other location on the screen without changing it's functionality. Obviously the possibilities for content interaction and presentation were limitless. Web design had never been so fun or rewarding.

So after the initial reveling was over, it was time to realize the benefits of the SVG standard, and deploy some content. Internet Explorer 7 was about to be released, I thought for sure it would come with some support for SVG, considering Firefox was already providing some support, and Opera was rocking near full compliance. Well, IE 7 came out without a lick of SVG support. Surely they would wise up, get with the program and provide support in a service pack? Wrong. Hell, IE 8 *just* came out and Microsoft is still pretending like SVG doesn't exist. Not even a hint of support for SVG. Why you ask? Why would Microsoft totally ignore such a great standard that offers so much promise for the web? Two words. Silver. Light. Yes, Silverlight and XAML. Microsoft basically rips off the SVG standard to define XAML, then turns it's back and pretends like SVG doesn't exist, again in order to promote it's own agenda. For some reason as I write this, I am reminded of Death of a Salesman: "You cannot eat the orange and throw the peel away. A web standard is not a piece of fruit!". Microsoft ate the fruit of W3C's labors, threw the rind away, and walked off.

This transgression by Microsoft is magnitudes more serious than the Bungee / Halo fiasco. This makes that seem completely insignificant. This is Microsoft using it's market share to try and KILL a standard. With the majority of end users using Microsoft's browser, they can pretend that SVG doesn't exist, and thus make the majority of the world not realize it exists. No one will push forward with SVG, because the world's most popular browser... Microsoft Internet Explorer stubbornly refuses to support it. The world is being denied an open web standard that can provide interactive and dynamic content on level with Adobe's Flash, but is natively supported in browsers. Can you imagine the community driven image gallery projects like Gallery, CopperMine and 4Images but based on SVG that would exist if Microsoft had supported SVG six years ago when they should have? I don't think you can. Hell, I'll make a prototype just to hint at what you are missing (as long as you aren't using IE).

At first it was denial... I continued to search the web for news that Microsoft wasn't being that obstinate, and actually had plans to support SVG in IE at some point. I hunted through the forums waiting for some Microsoft engineer to say in a forum somewhere that yes, of course SVG support was on the IE development roadmap. Time dragged on, and eventually I couldn't deny the facts anymore... Microsoft was doing this... HOW COULD THEY?!? AAARRGGHHH!!! The anger was the worst because I had little outlet for it. My friends didn't seem to understand the fury that drove my diatribes about how evil Microsoft was. I hoped that maybe... maybe if they didn't feel threatened by SVG they would relent and offer support for it. I cried a little. Tears fell from my ocular devices, shed for all the potential glory of SVG that would never be realized. I became sullen and discarded web development for years. My webpage lied in repose. It was during this time of depression that I rediscovered Linux.

I had used Linux before, but never considered it's real potential. I always used it when telnetting to shells, and back in 1996 I built a Frankenstein computer assembled from the corpses of other machines. I installed RedHat Linux (v2 I think) and used it to make a CGI application for the ISP I worked for. It was pretty neat, and I liked the feeling of being totally lost in a new operating system, but at that time I still had much to learn about windows, and only considered Linux as a second-rate OS. Back on the real timeline, I got myself a copy of Gentoo and again installed it in a Frankenstein computer, this time one that was assembled in a milk crate (real classy, eh?). For some reason this time around, my feelings for Linux were much different.

This change in feeling can be likely attributed to two things. One, I felt I had pretty much learned everything that wasn't hyper esoteric about Windows. I knew XP inside and out, I knew Windows server, all about Active Directory, Group Policy Objects, terminal services, etc. I had also used a crap ton of C/C++ Microsoft APIs most people had never heard of like the TS API and the GP API. Two, I had started to hate Microsoft and finding out that from within Linux I could emulate the fast majority of Windows applications uisng WINE and even emulate DOS using DOSBox gave me a great feeling of control and satisfaction. Furthermore, Linux was aimed at people like me.

People who wanted the freedom to customize EVERYTHING if they wanted. People who wanted the source code for all the applications they use just in case they wanted to modify how they operated slightly. People who wanted to work using formats that were standardized and open. People who didn't like being told how to use their computers. People who saw new computer technology and standards not as either a threat or a chance to capitalize, but as a road to innovation where they could potentially improve the quality of digital life for computer users. It is these people who embrace Linux and opensource, and feel oppressed when working with Microsoft Windows.

After this recent excursion into Linux, I approached the operating system as a replacement for Windows, not just a diversion from. This meant I needed to be able to do everything I typically did within Windows without inconvenience. After years of using nothing but Gentoo, Debian and of course Ubuntu, I can honestly say that I have found absolutely nothing I cannot accomplish in Linux with similar or improved efficiency, except perhaps playing TES: Oblivion.

I can take the lack of support for new PC games. Oblivion was a time vortex anyway... I stopped playing it on Windows before I really got into it. I knew otherwise I would sink countless hours into the void. It seemed a game that you could continue playing forever. My father actually proved this wrong though... he played Oblivion into the ground. He only stopped after he got 100% chameleon so nobody could see or hurt him. At that point he resorted to just hanging out on the front porch of his house in Anvil watching the people go by. Apparently this lost its appeal pretty soon.

Back to the point though, I try not to spend too much time playing games anymore. I started playing text MUDs back in like 1996. I hardly ever left the house anymore. My friends were pissed because I wouldn't join in the D&D campaigns we constantly ran. My parents were pissed because the phone line was CONSTANTLY in use and nobody could call without getting a busy signal. I was only pissed when Creeping Death, Age of Legends or Carrion Fields went down. Diku/Merc/ROM was the stuff dreams were made of. I would often have dreams about being on the text mud. My dreams actually consisted of text on a screen. It was absurd.

When the first graphical MUDs started appearing, I was ALL over that. I was playing the first real graphical MUD, Meridian59 back when it was only the town of Tos. A single large room. I beta tested and played the hell out of pretty much every significant (and some insignificant) MMORPG that came out. Beta tested for Meridian59, Ultima Online, Asheron's Call, Star Wars Galaxies, and finally Ever Quest. For some reason after Ever Quest, I abandoned MMORPGs. I haven't touched one since the EQ beta. I just decided to not sink my time into games like I used to. MMORPGs were becoming the ultimate distraction... to the point of absurdity. People were living richer lives in virtual worlds than they granted themselves IRL. I digress. Perhaps I'll pontificate about it at a later time.

For all my haughty talk, I still like a good distraction once in a while just as much as the next guy. Fortunately there are a scad of options for digital entertainment in Linux. Many of the best games for Linux are found under emulators. I play PSone games using ePSXe, n64 games using Mupen64Plus, SNES games with zSNES, GBA games with VisualBoy Advance, DOS games with DOSBox, and Windows games with WINE. There are plenty of options for gaming input devices in Linux. Any device that is HID compliant will likely run without problems. There are devices that will let you plug PS/PS2, NES, SNES, and n64 controllers right into your computer and use them with your emulators. There aren't any good n64 adapters on the market since the Adaptoid went out of production, but I'm planning to take the time to make a microcontroller circuit for connecting them. Maybe I'll share.

With no shortage of solutions for the biggest complaint about Linux (no games), I don't have any immediate shortcomings I can describe. Maybe its my rose-colored glasses. Maybe its the huge array of upsides clouding my judgement.

Linux can be crafted to be as flashy or as spartan as the user likes. You can choose between a number of window managers (Gnome, KDE, XFCE, Enlightenment). You can have screen widgets. You can customize every single icon. You can install custom docking managers like Avant Window Navigator or Kiba Dock. You can go all out and install Compiz (previously Beryl) to give yourself wobbly, burning, alpha blended, motion blurred, dynamically zooming windows that exist across 4 desktops on the surface of a transparent 3D cube that you can rotate under a custom sky dome. There is no question that the Linux desktop environment is far superior in scope, functionality and configurability when compared to Windows.

I started my foray into chaos over four years ago now, at the end of 2004. I stumbled across a book by Clifford Pickover and Elahe Khorasani in the local library. It was about chaos, fractals and computer graphics.

At the time I was working on a game engine in Direct3D, so I quickly created a templated complex number library and set to work visualising some systems. Nothing worked at first, I was just getting pure black images as a result. Then... I thought I saw something. I increased the contrast of an image, and sure enough there was *something* hiding there in the darkness. I tweaked and adjusted my code some more. Finally, this Julia set materialised out of the void.

It may look meager, and by all counts really *is*, but at the time when those chaotic curls first appeared on my screen as a result of a some complex math, I was floored. I carefully adjusted the boundaries and sampling frequency of the system, and effectively zoomed in.

The tiny crevices in the curls opened up and spewed forth more detail. Where would it end? I spent hours rewriting this recursive routine and recompiling in order to change parameters of this Julia set rendering, and soon realised that I needed a better way to interface with these complex systems.

I adapted my rendering engine and created a Fractal Playground, where I could explore and play in the infinite number of fractal landscapes out there. I was limited only by the speed of my processor and the precision of 64-bit floating point maths. To me, exploring these fractal landscapes was as exciting as going on vacation to some new place. Rewriting the governing equation, defining new complex math operations, adding and modifying terms... it was like a grab bag of free vacations. I never knew where I would end up next. Some of these places were drab and repetitive, while others astounded me as soon as I stepped off the tarmac. The ones that I really remember are the ones that were drab on the outside, but with a little digging revealed themselves as fractal geodes... containing an unrivalled inner beauty encased within a plain shroud.

This image which is aptly named 'Buried Treasure' I found about 10E30 deep into a fractal landscape that seemed totally composed of white noise. I followed just a hint of a crevice in the noise down to this treasure. There were no real discernible features until this image appeared. Quite beautiful as well... it looks like the fancy sort of wood carving you would see on antique furniture.

I moved on from the typical convergence digrams where the color of a sample was defined by the number of iterations required to push it's magnitude past a threshold value. It was possible as well to track the actual orbit of a sample in complex space as you iterated it. Instead of just looking at the magnitude of a sample as it was iterated, I started plotting the values on the complex plane.

I referred to these as vector diagrams. The first time I plotted the orbits, as with many things, it didn't seem to work or make sense. There was far too much going on to understand the nuances. I needed a reference, and to start with a limited set of points. I first drew a convergence map, and then plotted the orbit of each sample on every 16th row or something. The results were intriguing. I soon found that plotting orbits in an aesthetically pleasing and efficient manner was quite difficult. Many wild and very strange images resulted from my experiments with this temperamental routine.

There was much more to be done however. This playground was fine for exploring, and giving brief reports on these landscapes, but it vastly lacked "production" features. The images I saved were limited in size to my screen resolution. The playground had a fullscreen mode, and I resorted to using the windows PrntScrn functionality to copy the full screen images to the clipboard and saving them off that way. I knew it was possible to create fractal images of arbitrary resolution... sometimes I hard coded a routine to dump a huge image of a system. It was a clunky way of accomplishing things, and many times while exploring I wished for the ability to do this dynamically. I also needed a better way to interface with the complex systems. Bringing up a dialog box and tabbing through controls, or even clicking on menu items was an annoyance while exploring. Fullscreen mode and modifying parameters (besides the zoom and pan) were mutually exclusive.

I was a bit daunted at first at the notion of entirely reworking the playground. My main code module (the one that contained the Windows entry point) had like a hundred global variables and was just under ten thousand lines long. In my state of fractal fervor, every new feature on the playground, every little test and modification was implemented as a kludge. I only cared about immediate results, and had cast maintainability to the wind. I built an empire out of bubblegum and bailing twine.

I rewrote the core routines, and this time built them out of brick instead of sticks and mud. I encapsulated everything, and made the multitude of non-exclusive boolean options for the system into a bitfield. There was now a single "state" variable where each individual bit represented a switch I could flip for a given option. One of the biggest and most effective changes was implementing a physically-based camera that existed in complex space. Physically-based, meaning the camera had momentum and friction, and moving it was all based on applying forces. When zooming/panning the weight of the camera would carry it for a bit even after releasing the controls. Note that this pre-dated the weighted interface of the iPhone by quite some time, and this idea was purely my own.

Now with the interface exclusive from the OS, I had a platform for fractal exploration that I never needed to bring out of fullscreen mode. With input and update routines built much like a game engine, I could define esoteric keypresses to my heart's desire, and the only change would be in the class's UpdateInput method. As I thought of new options, I defined a new bit in the bitfield and added accessors. It was simple to add routines for dumping the current scene to a huge detailed image, or saving each frame to make videos from. Now THIS was the kind of vehicle I needed for traversing the world of complex geometry.

Around this time I was an active member of the electronica scene in my area. I talked with some of the people setting up shows and got myself an invite to be a VJ for an upcoming event. I secured myself a projector and hauled my behemoth number cruncher (thanks, Stu!) to the venue. I had myself a little electroluminescent keyboard that worked perfect for manipulating the system in the darkness of the drum & bass arena. People were definitely impressed. One dude recognized the Mandelbrot set and was amazed to see it in such a dynamic light. That was probably the biggest compliment of the night. Everyone thought the convolving text routine where I feed text into a vector diagram and let the system swirl it around was mighty badass. All in all, it was quite a success.

Bolstered by the positive feedback from my first VJ gig, I was determined to expand the capabilities of this new application. Around this time I started experimenting with strange attractors. I used vector vector maps in the original fractal playground to build a "hit map" where computed locations of sample orbits were added to two-dimensional array that far exceeded the depth of a typical image. After subsampling the complex field and running thousands of iterations, I ran some statistical routines to get an idea of the spread of this data. I managed to clamp this data into a range acceptable by images, and interpolated the data into this clamped region to get some visual output. The results reminded me of images of strange attractors I had seen before. I didn't really know what a strange attractor was, but I figured they must be scads easier to work with than these complex vector diagrams.

I created a new application (again using parts from my game engine) for visualizing and manipulating strange attractors. These strange attractors proved to be quite varied and appealing. I especially enjoyed the way they moved when modulating the parameters over time. I produced hundreds of high resolution images and a few videos of these systems, all which were received well. I needed to get this new form of chaos integrated into my VJ software so I could switch between the systems on the fly. It was around this time that I migrated out of Windows and into linux, so this application would be the first to be resurrected in OpenGL.

Well, this is it. I've become a blogger.

I spend hours every day wrestling with new software tools and devising new feedback systems and ways to exploit those systems. Some ideas that rattle around in my head are too grand to implement directly, so I just mull over them for a period, and then often forget about them entirely. I could fill a library with all the great ideas I've forgotten.

I thought that perhaps given a medium where I could get those ideas written down and out in the open, perhaps I would benefit from them more. In the process, why not share them with those who are willing to listen... perhaps they can benefit from them as well?

The blogosphere is an increasingly crowded space. As much as I would like to have one place where I can vet all my ideas, there is a need to hone my focus. Deviating from this norm once in a while should be acceptable, but in the interest of keeping ChaosEngineer coherent these deviations should be kept as infrequent as possible.

Thus this will be a place where one can come to find technical insight on using Linux, OpenGL and various other open source technologies. The general focus of this usage will be for the purpose of creating, interfacing with, manipulating or visualizing chaotic systems. Getting Linux to do what you want, and then streamlining it would be part of this process as well. I hope to expose plenty of generally useful information along this somewhat narrow road.