If you are generating files or folders programmatically every day, hour, a minute, or second… Then you will end up with a lot of files that potentially may take up a lot of space on your system. Space is not a thing we got unlimited of and due to that, it is often necessary to delete files within a directory that are older than a particular number of days.
I have often implemented this functionality in archive directories where the server should delete files older than 3 months due to GDPR compliance. Without performing automatic routine maintenance, these archive folders can get big and start to consume a lot of space on our servers or cloud storage. Instead of doing this manually (chance you will forget it big) we can create a very simple script that takes a few parameters and allows us to remove files to clean up and maintain the server to keep it healthy.
Create simple script to delete files older than X days at hardcoded path
The first thing we have to do is creating a python script (I’m using Visual Studio Code with the Python addon). Start by adding the shebang at line one (A shebang is a path to Python executables) and then import the following packages to perform the activities.
#!/usr/bin/python3
import os, time, shutil
Now we have to define how old the files or folders within the directory are allowed to be. This can be done by adding two variables, one for the time and one for the root path (the directory that we would like to maintain). Python is working with seconds when we are talking time, so in order for us to be able to make it into days, we would have to multiply the number of days by 86400 (this is the number of seconds for one day). This calculated value will then at runtime be subtracted from the current timestamp giving us the date for how old we allow the files to be. Also, remember to declare the path to the root directory. If you are on Windows, you can add “r” in front of the string in order to be able to use a double backslash (\).
Thirty_days_ago = time.time() - (30 * 86400) root = "/path/to/directory"
Now for the fun part – where we actually make the cleanup process for files and folders. The rest of the script will be relying mostly on the OS module that we imported in the beginning. This module will be responsible for listing contents within the directory, help us determine if the object is a file or directory, and delete files. The module called shutil is used for handling directories and in this script, we will use it to delete a directory. The script will be using a for loop to loop over each object within the root directory. Below is the code and I will be explaining what the different parts of the code are doing:
for i in os.listdir(root):
path = os.path.join(root, i)
if os.stat(path).st_mtime <= thirty_days_ago:
if os.path.isfile(path):
try:
os.remove(path)
except:
print("Could not remove file:", i)
else:
try:
shutil.rmtree(path)
except:
print("Could not remove directory:", i)
As you can see above the script is looping over each object listed in the root directory. It’s then comparing the creation time of the object to the timestamp we calculated in the beginning and then removing it if the if statement is evaluated to true.
Below is the complete script you can copy and use in your own environment. If you would like to be able to use this as a dynamic script, please check my solution below, where you can append arguments at execution time for time and path.
#!/usr/bin/python3
import os, time, shutil
# Declare how old files are allowed to be and set the root directory
thirty_days_ago = time.time() - (30 * 86400)
root = "/home/king/nextcloud_server_backups"
# List all files and folders in the root directory
for i in os.listdir(root):
# Concatenate the root path with the file or folder we got from the above listing method
path = os.path.join(root, i)
# Check if the file or folder is older than 30 days
if os.stat(path).st_mtime <= thirty_days_ago:
# If that is true and the object is a file, try to remove it
if os.path.isfile(path):
try:
os.remove(path)
# File could not be moved, show an error
except:
print("Could not remove file:", i)
# If the object was not a file, then it must be a folder
else:
# Try to remove the folder from the directory
try:
shutil.rmtree(path)
# The folder could not be removed, show an error
except:
print("Could not remove directory:", i)
Create a script that takes two arguments to delete files older than X days at X path
If you like me often want this to be running automatically carried out by a service calling this script, you would want to make use of system arguments. It’s pretty easy and straightforward to implement this nice feature.
Start by adding sys to the import at line 3, as shown below:
Then remove the “old” time and path declarations and replace them with the following two declarations:
docPath = sys.argv[1] days = sys.argv[2]
The two above declarations take one argument each. The first one allows you from the terminal to specify the path and the second one allows for specifying the number of days we allow the files to be within the directory.
Now add the following line. It takes the declaration from days and casts it into an integer in order for us to do some math and calculate the timestamp later.
days_ago = time.time() - (int(days) * 86400)
The only thing I changed in the main part of the script was to match the new names of my declarations.
for i in os.listdir(docPath):
path = os.path.join(docPath, i)
if os.stat(path).st_mtime <= days_ago:
if os.path.isfile(path):
try:
os.remove(path)
except:
print("Could not remove file: ", i)
else:
try:
os-rmtree(path)
except:+
print("Could not remove directory: ", i)
Below is the full script to copy and implement into your setup:
import os, time, shutil, sys
docPath = sys.argv[1]
days = sys.argv[2]
days_ago = time.time() - (int(days) * 86400)
#docPath = r"C:\Users\chs\Desktop\removeTest"
for i in os.listdir(docPath):
path = os.path.join(docPath, i)
if os.stat(path).st_mtime <= days_ago:
if os.path.isfile(path):
try:
os.remove(path)
except:
print("Could not remove file: ", i)
else:
try:
os-rmtree(path)
except:+
print("Could not remove directory: ", i)
Summary
In this tutorial, you have learned how to use the os module to list all contents of a directory and shutil to remove the files that are too old to stay in our directory. You have also seen how to use arguments and parse them to the script from the terminal. If you got any issues or suggestions, please let me know in the comments and I will try to help you. As always – Happy Coding!
