Integration projects aren’t like green-field software development projects. They are ugly, they involve systems and platforms you can’t control and you have to learn to live with finished product that will offend your engineering sensibilities.
Let me tell you a little story.
We outsourced fulfillment operations for one of our companies just about a year ago. We had no core competency in warehouse operations and fulfillment, so it made sense for us to partner with someone who had them. Overall the outsourcing project was a success, and our heavy sales season went smoothly this year. That’s the good news. The bad news is that endless hacks and compromises we’ve had to make in order to cobble together a working software platform.
Our vendor partner provides a software platform with a browser-based interface to manage orders and shipping and an application-programming-interface (API) for reading and writing data. The plan was to use the API to send orders from our customer-facing web platform to their fulfillment and shipping platform. And we’d use the API to get order processing information back from their platform to update the customer records behind our web platform.
Our fulfillment partner provided a production platform and a test platform. This would allow us to build out our API interfaces and push orders to the test environment to make sure that everything worked properly using real data and processes. This is exactly where the surprises popped up. First, the test environment did not match the version of the production environment, which is less than ideal. However, I’ve seen plenty of cases where the test environment is a release or two ahead of the production environment. This lets you test new features before you use them in production. Except, in this case the test environment was BEHIND the production environment. And we were planning on using features for our integration that were only running in the production environment. Just how in hades were we supposed to test that the code worked without having access to a live environment in the test system? (Vendor shrugs their collective shoulders). Now mind you, this company is well known and is successfully servicing several big well-known brands. This meant that we’d be going into the go-live weekend depending up on features we could not even test.
This wasn’t the only problem with the test environment. We could only load data such as product codes and inventory into the production system – not the test system. And we had to the load the data via flat files and ftp. Well, you can see what’s coming, can’t you? We loaded up a bunch of data into production, messed the formatting up, which then got replicated into the test environment. The vendor could only refresh the test environment with data from production once a week, on Monday mornings. Bugs were plentiful in the test environment, including one that duplicated all orders a dozen times. This made training the customer service department a total nightmare, as they did not understand why a single order showed up a dozen times or more. This problem did not appear in the production system, but we could not load orders into production, even though we were not live because there was no mechanism to EVER delete an order from production. And the hits just kept on coming.
We used the API to build code to check the status of orders, so we would know when they had shipped and could load the shipping information back into the web site so customers could track their orders. Oops, turns out the system only updates the orders a few times a day and our code was overwhelming their system checking statuses every half hour. Their solution was to FTP us a shipments file every night. So we got to throw out all of our shipment status code and build a file to parse and load flat files every night. The flat-file generator was limited to using ONLY commas as field delimiters, and commas are valid characters in our customer facing system. That meant that we had to get super clever with our field parsing code. In the ten months we have been in production we’ve uncovered numerous instances where the daily shipment file is missing records. The orders shipped and the on-line dashboard shows they have shipped, but they are missing from the daily FTP file. We didn’t notice this until we ran our first month-end close and found that the vendor’s shipment tallies didn’t match our records. After some exhaustive research on their part their answer is that they “don’t know why the file isn’t always complete”.
The list goes on and on.
It should not have to be like this. But it is it like this. Why? Well, the vendor is running multiple different enterprise systems in their shop, likely both a warehouse-management-system (WMS) with some enterprise-resource-planning (ERP) modules and a bunch of ancillary systems (address verification, Fedex, UPS, USPS, etc.). Plus, they’ve written their own custom code to glue these systems together and slapped an API on top of the whole mess.
When we evaluated them as a vendor we reviewed the API manual, which looked great on the surface . The sales team assured us that the test system was a mirror of production. Customer references surfaced no problems, even when we asked direct questions about the API and the software platform. Surprise, surprise – references LIE.
On our side of things we have some clunky legacy code underlying the web site. Our engineering team is slowly but surely refactoring the ugly stuff, but this work gets a lower priority as compared to any customer-facing work. Between our two companies, we’ve got six or seven major systems from at least four companies all trying to play together nicely. We’re getting there, but it hasn’t been easy and we’ve had to learn to live with workarounds that only sorta work.
Integration is messy. Remember that when planning out software projects that include integration to external platforms you don’t control. Underneath the covers you are likely talking to some ungodly Escher-like mélange of code. It will take you longer than you think, it will cost you more money and you are going to have to live with lots of unforseeable compromises.