A Brief History of (Wasted) Time
I've been at this Pokédex thing for a long, long time. I recently stumbled across fragments of old code scattered about my hard drive, in copies of backups of backups of copies from several computers' lifetimes ago. Much has been lost, in part because I didn't know what source control was, but I've managed to recreate a rough timeline of veekun's history.
Most of the version numbers below are assigned retroactively, as I only actually gave versions 3 or 4 their numbers at the time. (So, yes, there was nothing I called version 2.) Also, because the production code was updated more or less regularly, the notion of a "version" is a bit arbitrary—where I've had to invent version numbers, I've tried to put the split around major style and infrastructure changes.
The date ranges are extemely approximate and cover initial development through abandonment, so they overlap when I had some sort of beta or dev code in progress.
Untitled v1: November 19, 1999 – unknown
Covered: Pokémon from Red and Blue only. No move pages; no move descriptions, even.
Code: A single Perl CGI script. A baaad one.
This very first attempt didn't have a version number, was never made public, and never even touched the Web. The only remaining copies of it, if there are any at all, are on a 486 packed away in storage somewhere. I only even know the starting date for sure because it appears in the footer on a later copy.
Source control: Ha! I'd never heard of such a thing. At the time my options would have been about limited to CVS, so maybe that's for the best.
Data: A single text file, as may be expected from a 12-year-old beginning programmer. It looked something like the following:
1 Bulbasaur grass poison 2'4" 15.2 ... 2 Ivysaur ...
I have trouble imagining something more difficult to work with. I regularly had bugs because a line was missing, which meant that all following Pokémon would have their fields shifted by one. I had to read the entire file to get all of this information on every page load. And, of course, asking something like "which Pokémon learn move X?" was an utter nightmare.
TMs were stored as a string of fifty-five ones and zeroes, with each digit indicating that the corresponding TM could or could not be learned by that Pokémon. That is, Bulbasaur had something like '00100101110000...' because it could learn TM03, TM06, etc.
Moves were stored as '01-033 01-045 07-073 ...'. That's level, hyphen, move number, all padded to be the same width (for some reason).
Design: Unfortunately, I don't have a copy of what the output actually looked like, but I remember bits and pieces. Each page was colored to match the Pokémon's "primary" type, where "primary" was a notion I completely made up; basically, grass was "more" of a type than poison, and water was "more" of a type than ice, etc. The sections you know and love today were all tables with obvious borders, all colored with the "primary" type color. The page background was dirt sprite from the games, but with the dirt specks colored to match the type color. Possibly the height of 90s Web design.
UltraDex v2: late 2000 – July 2001
By now I had come across a glorious bounty, in the form of hosting from a buddy at my high school. I was now on The Internet, for real. And I frittered away this opportunity on convoluted JavaScript page navigation that has kept the actual Pokédex itself from being cached by anyone, anywhere. At least I gave it a cool name: "UltraDex". The ultra in dexes.
I do know that this brought about the first attempt at spin-offs, including a tool for finding the clock reset code for Gold/Silver and a little JavaScript memory game. I also started writing a walkthrough for Red and Blue (a bit late, I know) that was packed ridiculously full of links and cross-references and illustrations. I still think it was a good effort, even now; it's too bad I only got two badges in before moving on to something else.
Covered: Pokémon from Red, Blue, and (I think) Yellow. This iteration also had the first move pages and type pages. It even had dual-type pages, an idea I've never really resurrected: the page for water/psychic, for example, would show how much damage that type combination took from other types and which Pokémon were that type combination. This all required a much more sophisticated lookup mechanism, which I dutifully wrote—in JavaScript.
Code: Perl CGI, of some sort.
Source control: pokedex2.pl.bak.20000501
Data: Now I had wisened up! I no longer had a single massive data file. Instead, I had split every record into its own file and loaded them each into separate arrays. While this was still a horrible approach in hindsight, it was a vast improvement over what I'd had before.
Sadly, TMs and moves were still stored the same way. With the introduction of Yellow, I even had a separate 'y' value in the TM strings that indicated a TM could only be learned in Yellow.
Design: Good question. I have no idea. Pretty sure I'd reverted to simple black text on a white background, and I still have the frameset I used, but I don't know what the actual pages looked like.
At some point I also created what I called the "MiniDex": a tiny pop-up window crammed full of everything that the first incarnation had, with the sections becoming JavaScript-powered tabs. (In my defense, this was almost usable for the amount of information that was around in the R/B/Y days.) The tabs and backgrounds were, of course, colored by the type of the Pokémon.
UltraDex III: 2001 – 2002
Between my source control style of "make a copy and call it 'oldwhatever.pl.bak'" and lost modification dates on everything, it's difficult to pin down exactly when I decided to call it version 3. I even have documents that refer to it simultaneously as versions 3 and 4.
Covered: Pokémon, moves, and types from RBY and GSC.
Code: Perl scripts, spitting out static pages. (See below.) Navigation and pretty much anything dynamic became done by JavaScript that only had to work on IE5 and definitely looked like it.
Source control: Nope.
Data: The big technical landmark is that I migrated from flat text files to this shiny new thing called XML. I had this unbelievably massive 650KB XML file containing everything I knew about Pokémon, including the new shiny Gold/Silver/Crystal data that I'd painstakingly hand-copied from half a dozen other sites. An exerpt:
<poke pokeid="131">
<name>Ditto</name>
<jap_name>Metamon</jap_name>
<evolve>65</evolve>
<height>12</height>
<weight>9</weight>
<type1>normal</type1>
<type2>none</type2>
<kind>transform</kind>
<old_gameshark>4C</old_gameshark>
<cap_diff>23</cap_diff>
<breeding>DD</breeding>
<genders>255</genders>
<data>
<redblue>Capable of copying an enemy's genetic code to instantly transform itself into a duplicate of the enemy.</redblue>
<yellow>When it spots an enemy, its body transfigures into an almost perfect copy of its opponent.</yellow>
<gold>It can transform into anything. When it sleeps, it changes into a stone to avoid being attacked.</gold>
<silver>Its transformation ability is perfect. However, if made to laugh, it can't maintain its disguise.</silver>
<crystal>When it encounters another Ditto, it will move faster than normal to duplicate that opponent exactly.</crystal>
</data>
<area>
<red>Routes 13, 14, 15, and 23; Unknown Dungeon</red>
<blue>Routes 13, 14, 15, and 23; Unknown Dungeon</blue>
<yellow>Pokemon Lab; Unknown Dungeon</yellow>
<gold>Routes 34 and 35</gold>
<silver>Routes 34 and 35</silver>
<crystal>Currently unavailable</crystal>
</area>
<moves>
<move c=" 1" gs=" 1" y=" 1" rb=" 1">Transform</move>
</moves>
<stats>
<attack>48</attack>
<defense>48</defense>
<special_attack>48</special_attack>
<special_defense>48</special_defense>
<speed>48</speed>
<hp>48</hp>
</stats>
<rby_tms></rby_tms>
<gs_tms></gs_tms>
</poke>
TMs were now stored as tags like <tm_01/> within the appropriate parent tag.
As it turned out, parsing this whole thing and then picking out the bits I wanted on every request took some ridiculous amount of time like 26 seconds. (Running that same code now takes almost no time. Hm.) In the end, I adapted the CGI scripts to instead spit out static copies of all possible pages, which I then uploaded. This was the first caching mechanism on veekun, I suppose.
Design: The first I've managed to recover! It's actually from a copy of 4.0, but the design was the same in 3. Mmm, check out all that JavaScript I had to fix so it would actually run in current browsers.
The main color scheme for the next few versions was white-on-black, because that's more edgy and cool. The type colors were still center-stage. I believe this was the first layout that was anything resembling 'clean', eschewing borders everywhere for simple headers.
You can also see that I was using an icon collection I'd cobbled together from several sets floating around the Web. Remember, the in-game icons didn't come around until Ruby and Sapphire.
UltraDex IV: 2002
Covered: Pokémon, moves, and types from RBY and GSC.
Code: Perl CGI. Pretty much the same beast as before.
Source control: Don't hold your breath. Subversion was barely off the ground.
Data: The whole XML thing wasn't really working out, so I abandoned it entirely. (It seems I was years ahead of the industry—zing!) Now, before, I'd had some two dozen text files, each representing a field and containing one record per line, all loaded into top-level arrays. This was a pain to manage, so I had a better idea: I created a handful of files with many fields apiece, separated by tabs. It was effectively My First SQL, sans the SQL. Naturally, I managed to cock this up anyway, and created a bunch of global arrays named poke_$fieldname instead of using hashes. But it was progress, I guess.
Design: Unchanged.
Eevee's Pokédex 4.x: late 2002 – late 2003
I eventually dropped the silly name and kept gradually fiddling with the Pokédex, bumping the version number arbitrarily as I went: 4.1b, 4.2, 4.5.
Alas, it all came crashing down sometime in late 2003 (or early 2004?), when my host's hard drive bombed a second time, and then his server went down entirely. My free hosting was no more, and I was busy off at college tinkering anyway. My creations didn't see the light of day for a good year or so.
Design: Mostly the same throughout, although it did change slightly towards the end. I'm still not sure whether this was an improvement. Ripped out a lot of that JavaScript cruft, though, and what was left actually worked in non-IE (which, at the time, was pretty much Mozilla 1).
Eevee's Den Pokédex "5.0": August 2004 – October 2005
Finally, I'd found a new buddy-with-a-server (Racso) and set about putting the relationship to good use. I even had a (sub)domain now: http://eevee.racso.com/. I was almost like a real Web site!
Covered: Pokémon, moves, and types from RBY, GSC, and RSFL. Abilities linked to a search, which is almost as good as a real page.
Code: Perl CGI. By now, though, the code was being arranged gradually into modules, although it still did silly things like have dex.pl which required dex.pokemon.pl if you looked up a Pokémon.
Source control: Nada.
Data: MySQL! I had spent the timeskip well; I was now faintly familiar with databases, and set about porting all my data into one.
Being only faintly familiar with databases, most of the ad-hoc tables I'd had in plaintext files were moved wholesale into MySQL tables. That is to say: I now had a tbl_pokemon.movesrb column that contained exactly the same '01-033 01-045 07-073 ...' gunk as I'd started with all those years ago. But now I could query it, and query it I did, with brilliant and convoluted CONCAT() constructs to find all the Pokémon that could learn some given move.
Design: Went from black to a starkly different light blue I called "disjoint", in an arrangement I still quite like. (The main problem with it, looking back, is that it's entirely monochromatic and no other colors will fit.) This heralded the second major arrangement of the pages themselves, introducing the three-column arrangement that still persists today. I also first introduced the long-since-gone type textures in this layout, to compensate for no longer color-coding entire pages by type. Unfortunately, my icon set was now dramatically short on icons, and you can see I had to pad it with shrunken sprites until I could find the icons introduced in the new games.
This really didn't last too long before I got my own domain name, of which you may have heard.
Eevee's Stockpile Pokédex "5.5": October 2005 – January 2006
Covered: Pokémon, moves, and types from RBY, GSC, and RSFL.
Code: In March 2005, I bolted my existing mass of CGI scripts onto mod_perl, creating a mass of mod_perl registry scripts. I didn't really understand what mod_perl was for besides making Perl "fast", so I never knew leaving it in that state might have been a bad idea (and later paid the price in administration time). I certainly needed the speed boost, though, as I was starting to get users, quite the new phenomenon for me.
At this point, I still had a healthy dose of NIH syndrome. I wrote my own CGI query handling. I wrote my own form generation and handling. I wrote—and it pains me even now to say this—my own ORM. It was glorious.
This was the first incarnation to, eventually, use actual templates (via Template Toolkit) instead of masses of print statements—a gigantic leap forward.
Source control: Getting there.
Data: MySQL.
Design: Fairly similar to what I'd had before, except that I had opted to go with a horrendous sea-green-and-blue scheme. I'm pretty sure my monitor was woefully miscalibrated when I whipped this up. I did finally drop the ancient fanart icons for the official R/S icons.
veekun.com Pokédex "6.x": February 2006 – present
Covered: Pokémon, moves, types, abilities, and (sorta) locations from RBY, GSC, RSEFL, and DPP.
Code: Now we're talking. I started very busily moving all the code, file by file, into Catalyst controllers. Now I had a real Web application that was far less awful to hack on. I continued hammering on it, gradually adding features as time went on; there haven't actually been any huge breaks or overhauls in all that time.
Originally, this ran on mod_perl, but I got sick of that crap pretty fast and switched to FastCGI.
Source control: This is where recorded history finally begins: the first commit was a Catalyst skeleton in my old private Subversion repository. Revision 234 on July 15, 2006. And then it was all lost, because I realized I couldn't release the source code, because I'd committed a password very early on. So I started over with a new repository. But that subversion server is still running and cheerfully reports that the very first public commit was made on Feb 10, 2007, ringing in a new era in open-source Pokédex technology.
Data: MySQL, although I gradually untangled my old text-file clutter into something faintly relational. It's still not quite there.
First design: Still blue and still the same basic shape, but much darker this time around and without a sidebar. I called it "midnight". (It was actually created before I moved to Catalyst, but I'm pretty sure the two changes were very close together, so I consider them to be one migration.) I still like this design, too, but it had the same problem: too much blue, and impossible to introduce much of any other color. You can see I actually tried to avoid this problem with the menus and yellow header text, but in the end the extra colors just didn't balance the blue enough.
The dark-blue design saw the rise of names in URLs rather than numbers, and the rise and fall of my custom-made type icons. I spent so long on those. :(
Second design: On April 22, 2007, I merged in the current brown layout ("fluff") with which we're all familiar. You can, of course, see it live. This was the first major revision of the actual page layout since I'd resurrected the Pokédex in 2005. I suppose it could be called version 6.5.
I'd about exhausted my blue ideas by this point, and went for an Eevee-themed brown as (a) it's easier to make light brown than light blue, and (b) I could think of more colors that fit in with brown. I did manage to fit in some greens and reds, but I was honestly never too happy that it still came out monochromatic.
veekun Pokédex 7: November 2008 – present
And here we are now. It actually shames me to see that I've been working on the "new" Pokédex for over a year, as of this writing; I can certainly see why everyone says never to start over from scratch. It's been worth it, at least.
Covers: Pokémon, moves, types, abilities, locations, ...? from RBY, GSC, RSEFL, DPPHS, ...?
Code: Sadly, the new shiny Catalyst application was really just the old crappy code broken into chunks. The templates were a mass of logic. Tons of data was embedded in various places in the Perl code itself. The database still had columns containing lists of move ids. And, worst of all, I had failed at one of my original goals from back in 2005: to make it possible for others to reuse my code. Nobody could use the Pokédex parts without lugging the site along, and nobody could use the site without running the Pokédex.
I had some five years' more experience than when I'd written the bulk of this code, and it just wasn't any fun to work on any more. Perl was also gradually grating on my nerves; I had a job working with it now, and having had some Python experience, its rough edges were just tedious to constantly work around. I needed to start over for real, and I needed to do it in Python this time.
Having been briefly exposed to Pylons (along with SQLAlchemy and Mako), I set about rebuilding the past nine years with my shiny new toolbox.
Source control: Subversion was, similarly, not at all a joy to work with. Spurred by a similar migration at my workplace, I finally saw the light; the code is now split across my spline, spline-pokedex, and pokedex git repositories.
Data: Currently just SQLite on the server, but ultimately PostgreSQL. Three out of three; it's much more a joy to work with than MySQL has ever been. Data has all been loaded from scratch, too, and is causing far fewer headaches.
Design: I actively decided from the start that I needed to be able to use multiple colors—and that meant not having any single one color in the spotlight. I decided to make the content area black-on-white once more, and tried to strike a balance between blue (my favorite) and brown (the Eevee color) elsewhere. I think the result is the best I've ever done: headers are a soft brown, and subheaders/highlights are a cool blue. Special color-coded things like the stat table don't look out of place for the first time in years.
Code
For the curious: I do, in fact, still have code to run the black XML Pokédex, the queasy-sea-green Pokédex, and the dark blue pre-Catalyst Pokédex. It's all pretty bad. I might grab some choice chunks if anyone shows interest, but I can't post it all wholesale for various reasons and I can't be buggered sanitizing it.
Design
Here's every design I still have a copy of, side-by-side, so you can see the evolution of this monstrosity.
Originally written on January 1, 2010.
