Directory and file handling in Python for real-world Projects.

1. Introduction

When you are working for an internal project or a college project, file handling is not something you have to worry about, most of the focus can be given to the algorithm development part. But when you’re working in a real-world project where the data complexity can vary from nothing to infinity, proper file structure handling, writing clean and modularised code is just as important as the algorithm. The need for defensive coding is imperative. It should handle the exceptions, the multiform of inputs and the code should not break in any condition. That makes the difference between an algorithm and a proper solution.

When you’re working with a real client, the data you receive can have all types of complexities. One of the major challenges I have encountered is the complexity of the folder structure.
What I have observed is that a large number of data-driven projects follow a standard algorithm flow. We will have a large collection of input files which can be PDF, JPG, TIF, CSV, etc depending on the problem you are solving and the technology which we are using. They will be given to us distributed in folders, subfolders, sub subfolders, etc. We will be needing to process all these files with some ML/AI algorithm and save the outputs retaining the same folder structure. The general Algo…