Occam's Razor

Like many of us in the information technology field, my career has been about creating and maintaining information systems. The techniques and technologies used have varied. Until now the process has stayed essentially the same thing. The process is something like this. Get people to put data into a computer. Store it somewhere. Apply business rules to it by writing a lot of customized code. Then spit it out in the forms wanted by various data consumers.

Really, that’s it. It’s doing with a computer what people used to do in their brains. Computers just have the ability to do these things much more quickly and reliably. But of course you have to tell computers precisely what to do and the order in which it must be done. This logic is what we call code or software. While it has not made me rich it has kept me gainfully employed and enjoying a comfortable lifestyle.

There were classically a couple ways to get information into a system. The most often used method at the start of my career in the early 80s was to stick someone in front of a terminal and have them enter data into forms on a screen. They then pressed a key and off the data went, through the ether and into a database somewhere. But there are other ways to bring data into a system. In the old data processing days one popular way was to load big reels of tapes from somewhere else and read them into mainframe computers. Since then we found more efficient ways of recording some information in a computer. Bar code scanning is probably the best-known way.

Once the information is in the system it is scrubbed, processed, compared with other information and placed somewhere else. In other words, it is sort of assembled. An information system is a lot like a factory. Raw material (data) is dumped in at one end. Out the other end comes data on steroids: information. You know much more about something from the output of the system than you know from the relative garbage of facts that fed it. And this information is typically used to add value, such as to gain a strategic or competitive advantage.

That stuff in the middle between keyboard and printer was a lot of usually hand crafted code. At the dawn of my career it was often written in Fortran or COBOL. During the mid to late 1980s it was more likely to be in languages like C, Pascal or PL/I. During the 1990s object oriented programming languages gained ascendance. Instead of C, it was C++. Businesses that ground out client/server object oriented applications used development environments like Delphi or PowerBuilder. Data and the software used to manage these data began to merge into something called objects. Logically the stuff was still stored separately. But conceptually an object let us programmers get our brains around larger and larger programming problems. As a result we learned the value of abstraction. Our programming languages became more ethereal. It became a rare programmer who could actually write a binary sort routine. Instead we call APIs or invoked methods on software objects to do these things.

Toward the late 90s critical mass developed around the idea that data should be easy to move around. Businesses needed simpler ways to transmit data with other businesses. This was one of those “no duh” ideas that someone should have successfully ran with twenty years earlier. Okay, there were ideas like EDI, but they were expensive to implement. Instead with the Internet finally ubiquitous enough to use as a common data transmission medium, a data standard for the web emerged: Extensible Markup Language, or XML. In the process data became liberated. Whether a field started in column 8 no longer mattered. Tags describing the data were wrapped around each element of data. An externally referenced XML Schema acted as a reference to tell you whether the data were valid or not. Instead of writing yet another unique application to process the data, a generic SAX or DOM parser could easily slice and dice its way through the XML data. Using objects and modules built into the programming language of your choice it became fairly simple to parse and process XML data. As a result there was at least a bit less coding needed to put the system together than in the past.

The newest wave in data processing is called web services. Just as XML became a generic way to carry data with its meaning over the Internet, web services provides protocols for automated discovery, transmission and retrieval of XML formatted data. Figuring out the best way to do this is still being hammered out. Protocols like SOAP are losing favor for simpler URL based methods like XML-RPC and REST. We’ll figure out what works best in time. But equally as interesting as these web services technologies are the XML transformation engines now widely available. The XSLT (Extensible Stylesheet Language) specification, for example, allows XML data going or coming into a system to be transformed in an infinite variety of different ways. It can be something simple like converting XML data into a web page with the XML data embedded inside it. Or XML can be rendered into something more complex, like a PDF file or a MS Word Document.

But what does all this mean? The light bulb finally went off yesterday. I was explaining to a colleague at work why I wanted a system I managed to have web services. My team understood the value. Data and presentation could be wholly separated. With data in XML it could fairly easily be transformed with the XSLT engine of our choice into the format that we chose. The effect of this is to markedly diminish the actual logic needed to set up and maintain an information system. The big payoff? In theory, fewer programmers are needed and it should be faster and easier to manage information. But in addition the system should behave more reliably, since less code is needed to run the system.

For example the system I manage is what we in the computer business call tightly coupled. It works great. But it’s just a pain to maintain. The data of course is stored in a classical relational database. To get it out and present it to the user we have to turn it into HTML. Right now we do this with a lot of code written in Perl. Naturally we get lots of requests to add this and delete that and show data rendered like this. And so once again, as programmers have done for a generation, we perform major surgery on our code and go through extensive testing until we get the results requested. But since we are a government system in a non-sexy agency we are grossly under funded. So most of these requests go into a programming queue. Many of these great ideas will be abandoned due to our tightly coupled system and our limited resources.

So what’s really interesting to me about these XML technologies is that we should be able to put together systems much quicker once we have the architecture in place. In addition we should be able to make changes to our systems much quicker too. We could end up with systems that in the classical sense require little programming. This example on the W3Schools site shows how incredibly simple it can be to take data from an XML data store and render it as HTML. Once the XML schema is defined and the template is written in XSLT then rendering it can be accomplished in just a few lines of code. Of course this is a very simple example. But when I think about what sort of effort and time would have been required to render this same result in those pre XML and web services days I am a little awe struck. The productivity potential is sky high.

So I’m starting to wonder: do XML technologies mean that information systems will no longer require any crafting by programmers at all, but will instead be easily assembled? If so this is revolutionary. But the pieces seem to be there. On the output side of the system XSLT and an XML database work fine together at spitting out information in a useful format. There is little or no coding needed here to make that happen. But what about the input side? There is revolutionary news here too. Initiatives like the W3C XForms project are finding standards based ways to gather form data intelligently. We programmers should not have to struggle too much longer with HTML forms, embedded with Javascript client logic and server based scripting logic. XForms will handle the job in an XML way that will minimize coding and markedly reduce maintenance costs.

And so there you have it: all the components needed to construct basic information systems in a generic fashion are nearly in place. Simple data collection and retrieval systems — what I have been doing my whole career — could potentially be done using open standards and without writing a line of code. With an XForms editor we will draw our forms and push it out to browsers and other web-aware device. Input interface: done. Web services can be utilized for the automated data interchanges needed between businesses. To realize this vision may require putting a SOA (service oriented architecture) in place first. A good application server will be able to get the data and persistently store it without much coding. And an XML aware transformation engine embedded in or talking to the application server will take our template of choice and render it in the format and media wanted.

Will programmers no longer be needed to construct information systems? Not quite, at least not quite yet. Few applications are as simple as the one I suggested. And there are hosts of other variables to be thought through, including quality of service requirements that often require programmers. But I suspect that over time we will see that information systems will require fewer programmers. Instead the emphasis will move toward those on the system administration side. Database administrators will still capture and manage data, but they will also tune the database for rendering content in XML. Business rules will move more into the database or into rules engines attached to the application server. The result should be fewer programmers steeped in the mechanics of languages like Perl. Instead we can expect more time spent tuning databases and maintaining business rules. Form data will be designed in an XForms editors. We will use similar tools to render output using XSLT.

Time will determine whether I am seeing the future clearly. But clearly I am not alone since this is the whole larger point of XML technology. Companies like Microsoft have created packages like BizTalk just for this purpose. (Their Visio product is used to diagram the business rules.) It should get easier and become less costly to create and maintain information systems. And over time we can expect that systems will be able to exchange and process data with each other much more simply.