Website file limit
Posted: Tue Nov 28, 2017 1:20 pm
Hi all,
I was reminded during the UKB social event to take another look at just how close UKB is to the hard limit of 300,000 files imposed by the service provider (and which is, I believe, a software limit). I thought we had at least another year to go before having to take action, but found that we were at 295,732 files. In other words, we were going to hit the limit within a month. The only saving grace is that I was wearing brown trousers as this figure flashed up on my screen ... And so, I've had to bring forward the inevitable and have spent the last few days thinking about options and implementing a resolution.
First and foremost, NO FILES HAVE BEEN LOST. They are all backed up on my machine at home, on two external hard drives, and in the cloud.
The vast majority of files are a result of attachments to posts (usually images). Based on various discussions I've had, I believe that the long-term solution is for files to be hosted on the cloud (using, for example, Amazon S3) where the 300,000 limit won't apply, for UKB to save post attachments there, and for UKB to link to these from posts. However, implementing that will take some time.
In the meantime, I've archived (removed from the server) all post attachments up until the end of 2012. No posts have been deleted, however, since we still want these to be searchable for interesting info. when needed. The result is that you will find images missing if they are associated with posts before 2013. Following this "intervention", we are now at 249533 files, a total reduction of around 45,000 files. I will, of course, be monitoring things more closely moving forward. I'm now thinking about a mechanism for reinstating (even if temporarily) missing images; watch this space.
To ensure that we still have access to the personal diary content, with images, I've created a PDF file for all members with a personal diary for each year (2010, 2011, 2012) and a summary of these PDFs can be found at: http://www.ukbutterflies.co.uk/forum_diaries.php. This link is accessible under the "Community" menu, named "Personal Diaries (archive)".
As usual, all comments/thoughts welcome.
Cheers,
- Pete
I was reminded during the UKB social event to take another look at just how close UKB is to the hard limit of 300,000 files imposed by the service provider (and which is, I believe, a software limit). I thought we had at least another year to go before having to take action, but found that we were at 295,732 files. In other words, we were going to hit the limit within a month. The only saving grace is that I was wearing brown trousers as this figure flashed up on my screen ... And so, I've had to bring forward the inevitable and have spent the last few days thinking about options and implementing a resolution.
First and foremost, NO FILES HAVE BEEN LOST. They are all backed up on my machine at home, on two external hard drives, and in the cloud.
The vast majority of files are a result of attachments to posts (usually images). Based on various discussions I've had, I believe that the long-term solution is for files to be hosted on the cloud (using, for example, Amazon S3) where the 300,000 limit won't apply, for UKB to save post attachments there, and for UKB to link to these from posts. However, implementing that will take some time.
In the meantime, I've archived (removed from the server) all post attachments up until the end of 2012. No posts have been deleted, however, since we still want these to be searchable for interesting info. when needed. The result is that you will find images missing if they are associated with posts before 2013. Following this "intervention", we are now at 249533 files, a total reduction of around 45,000 files. I will, of course, be monitoring things more closely moving forward. I'm now thinking about a mechanism for reinstating (even if temporarily) missing images; watch this space.
To ensure that we still have access to the personal diary content, with images, I've created a PDF file for all members with a personal diary for each year (2010, 2011, 2012) and a summary of these PDFs can be found at: http://www.ukbutterflies.co.uk/forum_diaries.php. This link is accessible under the "Community" menu, named "Personal Diaries (archive)".
As usual, all comments/thoughts welcome.
Cheers,
- Pete