Where's That Bus?

As a result of our past work on WMATA’s successful Trip Planner application (www.wmata.com/), TeraTech is currently working on another major transit project.  The client is a major transit authority, and the ultimate result will be real-time bus data for its customers, available via web browser, cell phone or PDA.  Two separate sources of internal bus routing and schedule data must be combined and converted into a proprietary XML format to be used by another application.  It’s a huge batch process that runs only occasionally, when anything changes in the routes or schedules, and it must be 100% accurate and reliable.  The client uploads nearly 300 MB of data, some in a database and the rest as flat text files, and our system performs complex data transformations including merging, validation, and interpretation.  We load the data into a relational database, and then use complex queries to generate the XML.  Although the processing is done in parts and combined at the end, the sheer size of the finished output file makes optimization of code and memory performance a critical factor.  After performing this “magic” our automated system notifies the client that their XML data is ready to download and sent to the customer-facing application.