Fragmented Development

I've recently been reading the planets: Planet Ubuntu and Planet Web Security, in addition to my daily blogs. Planets are really just lots of blogs tied together into one RSS feed, so in essence I just started reading forty more blogs in two additional subscriptions.

In addition to that, I've been reading more ordinary blogs. I really like subscribing via RSS feed, and having interesting information delivered to me on a regular basis. On average, I'd say I read about thirty posts a day, of varying lengths, depending on the blogger. This is a nice, steady stream of information – not too much, not too little – until you miss a day. Thirty doubles and becomes sixty, which is way too much information for one sitting. Imagine if I miss more than one day?

While this daily deluge of information is keeping me up to date, and I'm learning more than ever before, it is also quite daunting to manage. I've found that sometimes a busy day on Planet Ubuntu can make me late for work if I spend too much time on reading. The more I read at work, the less time I have to devote to actual work. To balance things out, I've created a few rules I follow when keeping up on the blogosphere:

It will be there tomorrow.

Since I use an RSS reader, I have no reason to rush through todays blog posts. They'll still be there tomorrow, and the day after that, and the day after that if I really need them to. Letting your daily reading impact the rest of your day is nonsense.

If you're really behind, try and read a little out of your daily routine. I usually read just in the mornings, so if I'm drowning in information, I read a couple posts when I get home from work or before I punch out.

Read It Later

If there's a thirty page long blog entry, and it seems awfully good, mark it for later and move on. Odds are that you'll have some free time to kill someday, and that will be great for reading that epic entry on interface design, or how to configure a streaming media server using open source software.

If you try to cram how-tos and in depth articles into your daily reading, you won't get anything out of them. Information that requires that level of involvement is better left for later reading, or applied to a project when it's needed. Most RSS aggregators have the ability to tag items as “ToDo ” or “Later ”. Take advantage of that, and put if off until you can really benefit from it.

Skim The Planets

One thing I've encountered continuously on the planets I read is that some posts are not aimed at me at all. Whether it's a technical comparison of different methods of managing packaging repositories in Linux, or new heuristics for penetration testing web applications using Javascript and the DOM... I couldn't possibly be less interested. These are posts with extremely specialized information, none of which will ever be useful (or understandable, for that matter) in the foreseeable future.

I know it sounds cruel, and I know it's hard to do occasionally, but you need to delete these posts. Don't read them, don't look at them, just get rid of them. If the title of the post involves a function, program, technique or language you don't use and have no interest in, odds are you're not going to gain anything out of the post itself. As a blogger, I don't mind if the knitting community never gets into this blog – they're not who I'm targeting with my writing. I'm sure the penetration testers, packagers, and kernel hackers out there don't care that I'm not reading their latest tips and tricks.

Hopefully these techniques will help you keep your head above water, and stop you from becoming overwhelmed when dealing with all of the information available to us as part of the net. We're stuck forever keeping up-to-date, but you don't need to let current events take over your life.

Out of all the projects I've been working on, one issue continually crops up. That issue is site navigation, and it is the single most important part of your site.

I'm not mincing words here, because there is nothing, nothing more important than your site's navigation. You can have the greatest site on the entire planet, with the richest content and the most to offer to your visitors, and if they can't get to it then you're sunk.

One of the common fallacies of site navigation is organizing your site based on the hierarchy you see. We have an example of this at Warren County – we started life with a single, alphabetical list of departments as our site navigation. It made sense to us, because Health Services kept up the Health Services site, and the County Clerk kept up the County Clerk site. The essence of simplicity, right?

• Warren County
  • County Clerk
    • DMV
    • Passports
    • Pistol Permits
  • Health Services
    • Rabies Clinics
    • Public Health
    • Homecare

Well, yes, that works wonderfully – for us. For the rest of the world that would actually be using our site, this didn't help them one bit. Only the department name was shown on the homepage, so if they were looking for a passport application, they needed to know beforehand that the County Clerk takes care of passports, and then logically that web site would hold the application.

You need to take a look from outside, and see what your visitors see. Instead of being a county employee, I had to take the point of view of a citizen who didn't know anything about the workings of county government. We began by creating four general categories of "stuff" that we have in the county: citizens, visitors, business, and government. It's really just a loose arrangement: there are citizens interested in business, visitors interested in business, and everything on the county web site is technically about county government, but you get the idea. Then, we took the pages of each departmental site and sorted them into one of these categories. So instead of first seeing a listing of department names, they'd see the a list of things for citizens to do.

• For Citizens
  • DMV
  • Homecare
  • Passports
  • Pistol Permits
  • Rabies Clinic

This allowed them to get where they were going in one click. We're still halfway there at the county: for right now, we still have the departmental listing, and only show the top five or ten links from each of the categories. But the important thing is that it's a start, and we can gradually move to a more logical navigation scheme.

This all would have been much easier if we had designed everything this way from the onset. The important thing to remember is to put yourself in your visitor's shoes, and then make decisions from there.

One of the things I've been doing as of late involves subscribing to the RSS feed of any sites I find in my travels - that is, any feeds I think might provide insight or intelligence. One of the sites I've stumbled upon is Design Meltdown, a design-heavy web site that showcases good design by category (colors, design techniques, genres of page, etc). I'm trying to improve my design skills, so this gives me some creative fat to chew.

Another thing it's given me is a cross-selection of the good graphical designs on the web. This usually includes a lot of extremely bad technical choices: flash-only sites, monster downloads, table layouts, and more overly image-based sites than you can shake a stick at. Now I understand that some design schools incorporate being a prick about micro-managing your type into every site design class, but seriously. The content of your site does not need every word, every line properly kerned. Leave the text content rendering to the browser/OS and the user. I might like to read in 32 pt Courier, and who are you to tell me otherwise? Locking me into your artistic vision takes away my right to view content how I please, and at that point that's not a web site – that's you, gratifying yourself, in my web browser.

But I digress. While I've seen a lot of crappy web sites that just happen to look beautiful (If you're using a major browser, on broadband, with all plugins enabled, with 20/20 vision), I've also seen a tremendous rise in the presence of extremely well programmed, standardized sites. Not only have these sites incorporated stunning visuals and excellent design techniques common to sites created by people who have a background in art, but the code has obviously been put together by someone that is extremely proficient with standardized CSS and HTML. An excellent example of this can be found in Big Square Dot.

The Big Square Dot page is extremely well designed, but just happens to validate as standard XHTML 1.0 transitional. There are a lot of div tags involved, but the complexity of their layout warrants that. Besides, I'm of the school of thought where some non-semantic markup is necessary to overcome shortcomings in current CSS implementations. Also, all of the HTML inside the div tags are semantic, so there are no problems in my opinion.

This kind of page gives me hope that the web design field is progressing in the right direction. After years, standards are finally catching on and making their way to the design side of the fence. Thanks to the improved rendering in newer IE versions and wider adoption of other standards-compliant browsers, it may be a trend that continues to take hold. Good news for all net citizens!

The Google Analytics folks recently announced the ability to track downloads of documents with their service. The method provided involved adding a onclick attribute to the anchor tag of whatever document you would like to track. Every time someone clicks the link to download/view the file, the _trackPageview() function is called and registers a hit for that file. Pretty simple, eh?

Well, unfortunately, it's not a very clean solution. We're supposed to add a new onclick attribute for each file we want tracked? We're supposed to just embed the Javascript right then and there? Isn't there supposed to be separation between presentation, behavior, and content? Ugh, I can feel my markup getting dirtier already.

To avoid garbling up my code, I've created a quick script that adds an event to any links that point to a file (as long as it's file extension is on the list). It requires an addEvent() function, like the Rock Solid addEvent() by Dustin Diaz, but that should be about it. The script itself is written in JSON, so it should be able to co-exist with any other scripts you have. Here's how it works:

• The Extensions Array

  All of the file extensions that should be tracked are kept in an array, ga_file.ext. If you'd like to track all Powerpoint slide shows that are downloaded from your site, simply add the string 'ppt' to the array.

• Call the Initialization Function

  The initialization function, ga_file.Init(), is called on the page load. It does the majority of the work in the script, including the following:

  • Get all of the anchors present in the document using the getElementsByTagName() function
  • Check through each anchor, and see if the target is a file with an extension that should be tracked
  • If the anchor should be tracked, a 'click' event is added that calls the ga_file.Track() function

• The Tracking Function

  The tracking function, ga_file.Track(), is extremely simple. It makes sure the pageTracker object exists (this is the object Google Analytics uses for tracking), and if it does, it calls the _trackPageView() function to record the hit.

Of course, I don't expect you to take my word for it. Here's the link to the source files.

• ga_file.js
• ga_file_doc.js - With full comments

Feel free to use it, but leave the comment at the top with my credit/link please. Let me know how it works out; this is a very early version, and has not had any testing, so use at your own risk!

There has been tons of news recently about IE8. It's extremely early in the browser's development, so take it all with a grain of salt, but it has been announced that IE8 passes the Acid2 test, which means the support for standards should be huge leaps and bounds over IE7, and completely surpassing anything IE6 could have hoped for. In effect, very good news.

Shortly after, there was an announcement on A List Apart that set off a wave of criticisms and hurt egos in the community. Apparently, to get IE8's true standard-compliant rendering mode, you would have to insert a meta tag that sets a "X-UA-Compatible" value to specify which version of IE you'd like to render your page. In effect, news that wasn't as great.

However, I've decided that I don't care. I'm working on a layout for the tourism web site that's based on absolute positioning and floats, and if I could get IE to render correctly by coding upside-down in Latin, I'd do it. Adding a simple meta tag just doesn't seem to be that big of a hassle, especially when you consider how many CSS hacks we have to employ currently to get things to work correctly. In addition, there are ways to put IE8 into standards mode without using a meta tag:

• Use the HTML5 doctype when it's available
• Send the "X-UA-Compatible" value as a HTTP header, either through server configurations or server-side scripting languages like PHP

You could also put up with IE7 rendering, as that will be the default mode for IE8. Somehow, I expect that won't be the most popular decision, no matter how much angst is held against Microsoft and the IE team.

Do I like the fact that we have to make further concessions to the Microsoft monolith to get things to work? No, no I don't. I'm the first person on the anti-Microsoft bandwagon, to the point where I've switched operating systems to get away from them. However, there's one very important thing that should not be overlooked: this could possibly be the end of developers having to work around Internet Explorer, with a single added tag.

I also appreciate the IE team's desire to disrupt as little as possible with their next release. Supposedly IE7 broke a lot of poorly-designed web sites, and they are not looking forward to that kind of backlash again. This seems like a perfectly reasonable solution to keep everyone at least partially satisfied, while also not causing chaos on a large scale for people who don't do web work for a living. It's unrealistic to think that "Mom 'n Pop dot com" will be able to run compatibility checks with each new browser release, when many people are blissfully ignorant of the whole concept of browsers in the first place. IE8's rendering modes will keep designers happy, because they will have a standards-based option for their web sites, and the rest of the unwashed masses will be content because the "Internet Icon" (You know, the big blue E?) will still bring up their recipe pages without showing them just how lacking the site designer's skills are. They're worried about their lobster bisque, not the state of web standards. That's our job.

Recently, a lot of .gov and .edu web sites have been the target of SQL Injection attacks that seed their database with Javascript that does all sorts of nasty things, including targeting an old RealPlayer vulnerability. For all the gory details on this particular incident, I recommend reading the Web Security Blog over at ModSecurity.com for all the details, or the original SANS report of the attacks.

As web developers, the integrity of our databases and our web sites are our livelihood. Attacks like this can severely damage our search engine rankings and infect our customers – neither of which are good for business. These kinds of attacks were bound to happen, because of the lack of understanding concerning SQL Injection, cross-site scripting, and other modern threats that affect the web. So, instead of discussing the impact of these attacks or the specific methodology that was used, I'm going to discuss some ways to prevent this sort of attack from affecting your web site.

Defending Against Web Attacks

The first thing you have to understand is how these attacks work. Again, I'm not going to get into the details (there are many other fine sources for that, just Google it), but nearly all of them require a way to modify your page before a victim receives it. This can be achieved through inserting malicious information into forms or HTTP headers, and this is where the majority of defensive measures can be put.

Filter Input

If you're receiving input from the outside, you need to expect the worse. When it's coming in from the outside, input can consist of absolutely everything. It can be null, it can be paragraphs and paragraphs (especially if posted from a form – the post method allows a lot of text, and HTML limits can be overridden), it can be broken unicode input and it can be PHP, Javascript, and SQL code. Make sure to clean your input of any encoded characters, trim it or reject any input that exceeds the size limits you set. Remove any unallowed tags to prevent people from inserting scripts remotely or modifying the appearance of your page.

I tend to do most of these things in a loop, until there are no more changes found. This prevents malicious input from being specifically designed to stick around after the first pass.

Escape Output

If you properly escape any output that will be used in HTML or inserted into a database, many attacks will lose their effectiveness. If someone was attempting to end your SQL query by inserting a single quote at the end of their input, ending your query, and inserting a new SQL query of their own after that, properly escaping the data you insert into your queries would prevent their single quote from being interpreted by the database.

There are easier ways to safeguard your database operations, like using the PDO class in PHP or using prepared statements, but you should always escape input sent to a database regardless.

Escaping characters used in HTML like < and >, make inserting HTML tags into your document rather hard. If the sites affected by the latest SQL injections had been escaping their output, they would have not automatically inserted the <script> tags into the bodies of their documents – they would have simply seen the raw HTML show up in the text of their site. While still not very desirable, at least it would have prevented malicious scripts from being executed on their site.

Validate Data

If there is a phone number field on your web site, "Watermellons" should not be an accepted value. Incoming data with formatting standards, like phone number and e-mail addresses, should be validated against that format. Addresses, names, and other similar fields cannot be validated against this kind of format, but validating what you can will shrink the number of vulnerable fields in a given form.

Radio buttons, checkboxes, drop down lists, and other fields with a fixed number of options should only accept those values. Even though there's ordinarily no way to go beyond those choices in a basic web browser, attackers can easily manipulate your forms to send whatever data they like.

Validating form inputs using Javascript is a great idea, and saves time and bandwidth, but is very easily bypassed. Make sure you have strict data validation at the server level as well!

Enforce Low Permissions

If your web site uses a database to house it's pages, make sure the database user that retrieves them for viewing doesn't have write permissions for the database. SQL Injections cannot occur without write access to the database, so whenever possible a read-only connection should be used.

If a certain script (like form processing scripts) needs to be able to write back to the database, set up a database user that only has permissions to write to the tables it has to. Why should your form user, that only has to write to the form_input table, also have write permissions to the users table?

Since the security of our web sites is so important, I think I'm going to write more about it in the future. This gives a few pointers for starting out, but is in no respects a comprehensive guide. For more about securing your web site against SQL Injection and other related attacks, make sure to read up on Planet Web Security. They aggregate many of the top security blogs out there, and provide bleeding-edge updates on the state of web security.

I've been messing around with the options in LifeType, and I have to say I'm pretty impressed. The options that LifeType offers seem to suit me pretty well, although there are some behaviors that irk me a little. I believe that I'll chalk them up to my inexperience for now, and hope for the best.

There is one part of polish I'm procrastinating, and that's the layout. The default isn't all that bad, but it's not all that good either. Technically it's gangbusters - XHTML compliant right out of the box! The only validation error is the result of a single missing tag: a <p>, and it's missing from one of the posts I carried over. If only more of the internet's HTML was this beautiful... but the layout itself leaves something to be desired.

I looked around, and found that LifeType uses the Smarty templating engine. I've heard a great deal about this product, so I looked around the internet for a quick introductory tour. After some light reading, I dove into the source of the template for the default layout and started hacking it apart.

I haven't gotten very far, but on first impression, the Smarty templates seem to let you put in some basic functionality without going all out with full-blown PHP code. Not bad, not bad. We'll see how it works once I'm finished mangling it - keep an eye out for updates!

Alright, so it can be done. I have everything up and running, and I'm obviously able to post. The default layout is a little hokey, and I'm stuck using a WYSIWYG (creates HTML for you) editor. I'm sure these things can be changed, but I was just able to get the whole getup working on Thursday night.

It seems that LifeType is, indeed, W3C compliant. This was a requirement, and not easy to do with generated content. Nice job, LifeType folks. My only real complaint was the 1,600 files it requires to run. I'm hoping that I can prune off some of that, like the MP3 player flash component.

Overall, I think this might be a satisfactory blog choice. We'll see how hard it is to port all of my previous blog posts here, and then we'll talk.

Edit: After a whole lot of copy and paste, the new blog has all of the old entries copied over. Comments were lost, unfortunately. I'll get back to actual posts right after this... after I figure out how to make this look less 'default'.

Due to some discussions I've had with a friend, I think I may be leaving Blogger for greener pastures.

My own web site, FragDev.com, has seen some terrible neglect on my part. I believe the last "news" I posted there had to do with the work I put in for a local political candidate at the beginning of his campaign. Unfortunately he lost a couple of months ago, and his blog's been deleted. If the subject of news ceases to be, does it still qualify as news?

Anyways, I'm going to be trying to set up my own blogging software and see how that goes. I've toyed around with writing my own, but that seems like way too much trouble for what it's worth. The posting bits would be easy (I write all my posts in HTML anyways), but comments could potentially be a beast to work out. Throw in elements like RSS feeds and I'm starting to think that downloading open source blogging software doesn't sound like a bad deal.

While I've enjoyed blogging with Blogger, I think I'd like more control over my blog. I'd also like it if I had a neat-o web address like "http://blog.fragdev.com" or something akin to that, since I've been moving my blog and my business closer and closer in theme and purpose. Bringing a little pagerank to the fragdev-dot-com homepage couldn't hurt, either. Whatever I do, I'll leave a link here when I get everything up and kickin'.

From the moment I was introduced to the open source movement and free software (as in speech*), I've tried to embrace it. For a web developer, free and open software seems like a perfect pair for the free and open nature of the internet. I've grown accustom to looking under the hood of any web site I come upon, so why should the browser or operating system I use be any different?

If free and open software makes sense to you, or is even something you're interested in, this is a very exciting time to be alive.

Free and open software is just getting it's start. There have been rumblings from the movement since the early nineties, but just in the past couple of years has this idea really taken off - largely due to the opening Microsoft left by letting IE6 stagnate for years and years. Enter Firefox, a wonderful open-source browser that you could download for free. Safer than IE, better than IE, with plug-ins that made it gangbusters. Now, if you're a web developer worth your salt, you have Firefox installed with the Firebug extension.

These days it's not uncommon for someone to deviate away from the Microsoft way of life. Macs are becoming hugely popular among old UNIX admins, newer designers and programmers, and even mom-and-pop types that just want to have a computer that works – and works well. You know what Mac OSX is based on? The open source XNU kernel, which has elements of an open-source UNIX variant built in.

But what really makes things exciting is Linux. The Linux operating system is completely free, can be installed on whatever hardware you have lying around, can be customized for absolutely any purpose, and kinda works. Sorta. Most of the time. But that's what makes it so exciting!

If you're using open source, you're on the frontier of computers. It's like the original days of computing, when the corporations weren't making the innovations, they were being made by ordinary guys and gals with brains and too much spare time. You could be the next Steve Wozniak, or work with Linus Torvalds. Free and open software allows you to make a difference in the world of computing. Seriously, how many other chances are you going to have to make your "I changed the course of modern computing" story to tell you grandkids?

After I wrote my post about using internal links, I discovered that my original technique of using anchor tags is far out of date. Modern technology has given us a much simpler, much slimmer, much snazzier way to link to your content, and it's seen cross-browser support for years.

Recently, I was re-visiting an older page that used the old <a name="myName"></a> syntax, and decided to update it. However, the internal link was nested inside an H3 tag, so I tried this:

<h3 id="myName">Header Text</h3>

It worked like a charm. I quickly loaded up the oldest browsers I could find, and found that this syntax has been supported in IE since version 5.0 at least, and the only browser I had installed that didn't support linking directly without an anchor was Netscape 4.8.

So, if you're worried about your burgeoning Netscape 4.8 user base, I suggest you stick to the old anchor method. Otherwise, a good ID will do ya.

Sorry I have not been more attentive here at the FragDev blog, but this December is turning out to be a killer! For more details (that I won't bother pouring out here), feel free to check my personal blog on Xanga.

My PC is five years old, and starting to show it's age. I built it during my senior year of high school, piece by piece, with extra money I saved up from my pizza delivery job. I bought the more static components first (pieces that wouldn't be outdated as soon as I bought them), like the case and the fans. Slowly, I worked up to the main components, and eventually I had a full computer.

Somehow, it's lasted this long and has handled every program I have thrown at it, but like I mentioned earlier, it's time for an upgrade. While I probably could upgrade the machine itself, I think I'm going to leave it just as it is. I can always use the extra PC, and building a new PC from scratch sounds like too much fun to pass up.

Don't get me wrong – creating a computer from the various components is much more work than simply going to Dell and buying one. Also, building your own computer does not save you as much money as it did "back in the day". All of that aside, building your own computer is worth it. You develop such a strong understanding of hardware specifics, and how each affects the performance of your machine. After a while, you get a feeling what really makes a difference to your personal experience. For instance, my computers really serve me well if I overload the RAM and get a powerhouse video card - primarily because of my love of video games. Hardcore programmers might find that the processor and FSB make more of a difference, where multimedia editors might want to focus more on the speed of their hard drives.

Next time you feel your system is showing it's age, think about replacing it with a computer you build yourself. The best way to learn about computer fundamentals is starting from scratch and working your way up, and it's a very enjoyable process – Once you do it yourself, you never go back.

I've been messing around quite a bit with my Linux installation (I'm currently running Kubuntu Linux 7.10) because I'd really like to stay away from my Windows partition as much as possible. A big part of Windows' necessity for me has to do with my crippling addiction to games.

One of the ways to bridge the gap between Linux and Windows games is to use Wine, an implementation of the Windows API that runs in Linux. This essentially allows you to run Windows programs in Linux, with a few caveats.

I've gone through two experiments with running games under Wine: Valve's Steam games, specifically the 1st-person puzzle game Portal, and a great independently-developed space-based shooter named Transcendence (think Asteroids-style navigation with an in-depth space travel sim behind it). I've had quite different experiences with each of these titles running under Wine.

Transcendence was by far the easiest to get up and running, which was a big suprise. It turns out that this application doesn't tie into the registry or register itself with the Windows installer, and instead is basically just a standalone executable file. This makes it much less complicated to run under Wine. All that was required to open the game in all it's glory was for me to change to the Transcendence directory, and open the executable using Wine:

wine transcendence.exe

Presto, I was shooting around the galaxy in no time. To make sure that my Windows environment saved to the same location as my Linux environment, I used a symbolic link to share the directory across the file systems. Sounds complicated, but it boils down to using one folder in two places. Here's the command I used:

ln -s /[windows partition]/transcendence/ ~/.wine/drive_c/transcendence/

That was really it, too. No problems, no wacky configurations or switches I had to research, it just worked. The developer had mentioned that he had attempted to get away from Direct-X to allow this sort of thing, and I have to applaud him. While the game does run a little bit slower on my PC without Direct-X, it's not noticeable. If he ever provides a way to donate money or buy Transcendence, sign me up.

With Steam and Portal, however, the process wasn't as smooth. First of all, the Steam installer is distributed in a .msi file now, which is Microsoft's proprietary installer. This makes it a little harder to install using Wine, but not impossible. I found a great how-to guide for installing Steam using Wine on the ArchLinux Wiki that helped me out with a command that would run the installer:

wine msiexec /i SteamInstall.msi

Not bad, eh? With that, Steam installed just fine and ran without a hitch. The problem came with the games.

Since each Steam game is pretty hefty, downloading each one again over my internet connection just didn't make any sense when I already had them installed on my Windows partition. I tried the symlink trick, but Steam encountered problems when trying to run after then. I quickly scrapped that approach.

Since Valve is also interested in not making me re-download gigabytes of data from their servers, they have provided a neat feature of their Steam client, called "Backup Games", that allows you to back up your game data to your hard-drive. I backed up a couple of games I was interested in trying out in Linux (13 GB for Half-Life 2, Portal, Team Fortress 2, Half-Life 2: Episode 2), and restored them using Wine. Not expecting much, I ran Steam and tried launching Portal.

It worked. I was pretty shocked, but did notice that there was a huge performance hit. Apparently my hardware is very Direct-X friendly, because Portal dropped down to around five frames per second when running it under Wine. Not playable at all, but everything seemed to work. A small setback for such a major victory.

For now, I'll be playing a lot of Transcendence until I can scrap together enough money to build a new PC. My current computer has served me well, but it's showing it's limits, and I think it's time for an upgrade. I'm going to try and upgrade to 64-bit Kubuntu, to see if I can eek some more performance out of my AMD64 processor. But with all things considered, I'd say that Linux gaming may finally be something we can take off our wish lists, if only to replace it with "Native Linux games". But let's not get ahead of ourselves. :-)