Maximum Amount of Files

[Viscus]

Hi all

Last week i received a mail from Bluehost Support about to many files on my account. It was over 1 Million.
One reason is probably my uploading of my mail archive with a lot of mail lists.
The other could be my gallery installations. At the moment there was no problem to remove my archive on this account because i had at the moment a linux machine working at home behind an ADSL line. But i would remove those machine and this was also the reason to move to bluehost with a lot of space available.
So for the moment its not easy to monitor the hole amount of files placed on server. And if each mail will be saved as a file there will be a problem. But the same could be in future with the gallery installations if you have a big amount of pictures.
Had someone the same problem?
Is the only solution to use a root server with dedicated hosting?

I don't want make any problem with my account but it seems to be a gap between the amount of space and the max allowed (unwritten) amount of files. I know the amount of many little files is a problem for backup.
To reach the 1 Million File amount its only necessary to upload 300kB Files to an 300Gig Account.

Probably there is for the moment no solution for this problem.

Regards Adrian

[verifone411]

Why not create folders a-z and separate them by the first letter? This should be pretty quick and easy to do especially if you have SSH installed.

[Early Out]

How would that result in fewer files on his account?

[IvanAkimov]

are you using all 1 million? can you maybe gzip at least half a mil or so?

[verifone411]

I dont believe the number of files on his account is so much the problem as the number of files in a directory.

Since you are dealing with text objects why not store them in a mysql database for easier handling?





How would that result in fewer files on his account?

[Early Out]

I dont believe the number of files on his account is so much the problem as the number of files in a directory.That's not what BH told him.

A few weeks ago, one server suffered a corrupted file system, and it took an ungodly long time to get it restored. One of the reasons, according to the BH folks, was that there was one account-holder on that server who had millions of files. As a result, the restore operation took something just this side of forever.

[alligosh]

The issue is the reiser filesystem, which about half the servers use. When there are filesystem issues (and yes, ALL filesystems occasionally have issues) then the recovery in reiser is a function of time. Every million files adds about two hours of repair time under reiser. The box referred to had a user with twelve million files, which added about twenty-four HOURS to the repair, and could not be bypassed. We have since gone back to ext3, but converting the existing reiser systems is not currently in the cards.

And yes, archiving those hundreds of thousands of small files on a regular basis (aka, nightly for daily uploads to not slam the server making archives) would be acceptable in most cases to get past the max file number issue.

[TheWebHostingHero]

With such a huge amount of file, I have a hard time figuring why you have a shared hosting account instead of a dedicated server.

[verifone411]

1500 GB = 1572864000 KB

1572864000 KB / 1000 KB Files = Would still reach the max 1572864


Conclusion: More people will have this problem in the future I would imagine if they ever come close to using all the space.

I am at 64193.52 MB which I think is alot, but perhaps there are many others close.

[Viscus]

The box referred to had a user with twelve million files, which added about twenty-four HOURS to the repair, and could not be bypassed. We have since gone back to ext3, but converting the existing reiser systems is not currently in the cards.


Thanks a lot for your explain of the problem. I understand that this could be problem, but there is a problem with space available and to many files.

Regards Adrian

[Viscus]

With such a huge amount of file, I have a hard time figuring why you have a shared hosting account instead of a dedicated server.

:) I had now for a long time two linux boxes at home behind my ADSL Line. But i wanted to move to web hosting to a solution with less system administration for me. I need a lot of filespace but for the rest i don't had the assumption about a problem with the amount of files which could be a problem.