Max numbers of users?

In communications I had with Boonex pre-sales it was suggested that Dolphin 7 does not have the same "max number of users" limitations that Dolphin 6 had (which is a limitation based on the Linux file system's ability to hold more than 30,000 files in a single directory). While migrating data to D7 today I've learned that a cache file is created for each individual user. My current non-Boonex site has over 45,000 users. Once 30,000 of these users have logged in to D7, won't the max number of files limit be reached in the cache directory?  What happens at that point?  Will the D7 throw errors because it is unable to add files to the cache directory?

If this limitation exists, I guess I could run a cron job nightly that will delete the oldest user files and only allow the most recent 20,000 to remain. I'd rather do that than disable the cache altogether, but what I'd really like is for me not have to do anything at all.

Any feedback would be helpful...

I don't use the dolphin caching. I have had noting but problems with it. I have also not noticed much of a difference is speed with it on or off.

You might want to look into the deflate option for the .htaccess file. Let your apache webserver do compression to speed things up. There is a topic in the forums here somewhere on it.

There are other advantages to the dolphin caching that the deflate option will not take care of. The caching reduces the number of times dolphin has to access the database, so it does decrease the load of the database server.

But if need be, on a very large site, the database could be placed on it's own sever seperate from the webserver. Even clustering it if needed.

Thanks for the suggestions again Deano. I've already applied the apache deflate option that I read in a previous post and it does a tremendous job of reducing file sizes.

Regarding the cache, I think if many users access the site simultaneously there's going to be a degradation of performance if I have the cache off since the database calls get queued and file system calls do not.

Also, your idea to cluster the server is good but I don't have the budget for that, so I'm going to try to leverage the cache as much as possible.

I had to do some research. I could not believe that 45,000 file limit. And it's actually not really true. Performance drops the more files that are in there.

From what i have been able to determine, it all depends on the type of file system in use. ext2,ext3, RiserFS, ect. And if file indexing is on or not.

I could not find a percise answer on this one. So i question the accuracy of that 45,000 file limit.

I checked my file system type again and its ReiserFS, so there are no issues since ReiserFS doesn't use a fixed formula for computing inode locations, so I'll never reach the max files that Ext3 has (which for some reason I though I had). So this limitation would only concern those who have Ext3, which is good to know in case anyone here encounters this.

Thanks!

i can confirm its the file system that has the limitation. if you are running 64bit then you dont have to worry about this, and the concern that i knew about had nothing to do with caching, it really had to do with the number of members, since each member has a directory created, and FYI, we are intertangling file vs. directory under one tree having to do with the number inodes its capable of creating on a per directory implementation.

For the ext3 filesystem "There is a limit of 31998 sub-directories
> per one directory, stemming from its limit of 32000 links per inode."

"The ext2 inode specification allows for over 100 trillion files to
> reside in a single directory, however because of the current
> linked-list directoryimplementation, only about 10-15 thousand files
> can realistically be stored in a single directory. �This is why
> systems such as Squid (http://www.squid-cache.org ) use cache
> directories with many subdirectories - searching through tens of
> thousands of files in one directory is sloooooooow."

source: linuxforums.org

It seems that ext2/ext3 filesystem has a limit of 32,000 links in an inode, which in turn limits the number of directories in a single place to 31,998. I need to be able to work with a filesystem with no limit or a limit which is configurable. Is there such?

answer to above question from same forum:

Maybe reiserfs, I heard it has "dynamic inodes". That will keep reiserfs from running out of inodes, and maybe this "links in inode" problem too.

If not, perhaps XFS can do it.

source:http://www.issociate.de/board/post/487557/Files_per_directory.html

As far as I know there's no limit to the number of files in a directory
currently in ext3. There IS a limit to the number of files (actually
inodes) in the whole filesystem, which is a completely different thing.
According to Wikipedia "If V is the volume size in bytes, then the
default number of inodes is given by V/2^13 (or the number of blocks,
whichever is less)." There's also a limit to the number of
sub-directories in a directory, currently 32000.

xt3 also has a limit of 32000 hard links, which means that a
directory can't have more than 31998 subdirectories.

However, the original poster wasn't asking about hard limits, but
efficiency.

If the filesystem wasn't created with the dir_index option, then
having thousands of files in a directory will be a major performance
problem, as any lookups will scan the directory linearly.

Even with the dir_index option, large directories could be an issue. I
think that you would really need to conduct tests to see exactly how
much of an issue.

so the only solution would be to move into a 64bit environment or set your file system as ReiserFS, ext2 and ext3 have the sub-directory limitation

Sorry for the drawn out post, but if news like this is shared, it needs to be shared in its fullest. and i dont understand why boonex is not stating the need for a 64bit environment or a specific file system on the machine.

Regards,

DosDawg

Thanks DosDawg for your excellent analysis. I'm a big fan of your posts BTW

Only time will tell about the performance of a theoretically linear scanned directory vs. a database query. My bet is that the directory scan will be faster since the OS doesn't queue requests like the database does, so under load the database performance has a tendency to go south faster than the OS calls. Also, I think the directory contents are really not read in the same manner they are displayed, so it's not exactly linear. Who knows though, maybe it's the other way around...

Either way, I'll keep everyone posted once the site goes live and I see how the performance plays out.

Mucho thanks!

YAY! i gotta fan <^n3=0to>

yeah bro, thats a subject way too deep for this community i can assure you. hell we got folks thinking they can still run a production site from a lamedos machine with a dial up connection from home, and they expect 2m hits a day. oh yeah rock on.

Regards,

DosDawg

also on this, you are going to have to move away from mysql, as mysql cannot carry this kinda load either, and there is no way to backup a database without possibility of corruption in the process (once it has achieved proportionate size) . so you end up having to go to a commerical grade database environment, where the variables and direct table schema provide for somewhat more flexibility, and also allow for the population of a much larger database without the same noticable performance depletion.