Last Wednesday I checked my inbox and came across this message.

It was from one of my readers:

“Hi Adrian, I’m looking for a project work plan for a data warehousing project for the ___ ORGANIZATION ___ and need some guidance and document. Thank you”

Quite a broad request.

So I asked the guy for details.

The guy had come across my project playbooks, and was looking for a project plan for a data warehouse project.

“Can you give me more details?”

His answer:

“It’s a project for data warehousing a legacy system. The front-end application is scheduled to be decommissioned on January 31st, 2023. Post decommissioning, users must be enabled to retroactively edit data in either the legacy system or the new system.”

A decommissioning of a system! How interesting. Now we are getting somewhere!

A few more emails went back and forth.

I wanted to understand how I could best assist the dear follower.

Then it became clear he was basically struggling with getting started.

The system shutdown date had already been scheduled, and the IT guy had only 4 weeks left

Essentially, the guy had to 
(1) Make a plan
(2) Perform the decommissioning procedure in a safe way

Quite a challenge, right?

Then I started reflecting on my professional stages to see what project I had managed that was similar to this one. Having worked in an enterprise IT environment for over 10 years, I had been part of quite a few projects. But a system decommissioning …. wait!

In the shower, it hit me.

One of my projects was rolling out an Enterprise Content Management platform, which included a data migration from an existing “soon-to-break” archiving system including subsequent decommissioning.

The decommissiong involved extracting these documents from the legacy system:

• 25 million customer invoices
• 15 million sales order confirmations
• Around 6 million purchasing documents
• A few other document types with around half a million files
• All documents were stored in PDF format
• The legacy system was running on a Windows server and used an Oracle database for storing its data, so it was fairly easy for my system admin to backup the database.

As you can guess, these were important documents! If my company was getting audited by the tax authority, we had to show the original invoice for a particular sales order as a proof that this was a valid transaction. You can’t simply re-print the invoice if you cannot find it in your archive, because this would not be the original one. We couldn’t afford to lose a single document as part of the decommissioning!

Anyway, I digress the more I think more about this project, it was quite emotional.

Overall, the data takeover went pretty smoothly, but it took longer than we had estimated.

Here is the technical process we used:

• We used three scripts to handle the data extraction and migration to the ECM solution
• Script #1 would extract the PDF file and the corresponding metadata from the source system.
• The script created one ZIP file for each document record,  with each ZIP file containing a text file with the metadata like filename, creation date, document type and the actual PDF.
• The ZIP file was placed into a transfer folder from where another script #2 would pick it up and extract the content into an import folder of the ECM solution.
• Script #3 ran inside the ECM solution and its job was to correctly import the PDFs and the metadata into the ECM platform.

The key challenge of the data takeover was to ensure that each PDF was properly tagged so we would later know what type of document it was and which organizational entity it belonged to. We also had to ensure all metadata was migrated correctly. Otherwise it would later become difficult to find the right PDF based on, let’s say, the invoice number.

We did the data takeover take longer than expected? And what were our learnings?

• Remember we were dealing with tens of millions of documents that had to be backed up.
• The legacy system also was quite old, and a lot of administrators had put their hands on it, often not really knowing what they were doing. As a result, the system configuration wasn’t always consistent or complete, which caused problems in the data extraction.
• Never having had a real system owner and with no defined usage guidelines, the (legacy) system had basically been “abused” by the end users from Finance, Sales and Purchasing (they were not to blame, of course).
• People had uploaded all kinds of file formats to the old archive — anything from Word files, TIFF files, GIFS and JPEGs to Excel filees, and not just PDFs. This meant we had to figure out how to convert those non-PDF files into PDFs.
• File naming was another issue. Document files in the legacy system did not follow any kind of naming convention (the system didn’t enforce any). However, for the target system we had established clear file naming standards, like customer_invoice_<org>_<invoice-no>. My developer had to build another script for the renaming of files. Some files even had to be renamed manually, because there was no identifying data in the filename.
• The other challenge for the file takeover was to find the optimal batch size. We migrated the historical data in batches (x number of files per run), and it took us a few attempts to find the idead batch size for our setup. In the morning, I would check the status of the batch run and see if the files had been transferred correctly or if any manual intervention was required.

These were the main challenges we faced in the data takeover and server decommissioning.

Fortunately, it all went well and we were able to shut down our old archiving solution as planned and terminate the contract.

As you can see, a server/system decommissioning can become quite complex.

It’s usually more complex than we assume.

Based on what I have learned from this project, and from people having done similar projects, here are some tips for you if you’re preparing for a system decommissioning.