how does kernel handle new file creation

I wish to understand the way kernel works when a user/app tries to create a file in a directorty.

The background - We have a java applicaiton which consumes messages over JMS, processes it and then writes the XML to an outbound queue+a local directory. Yesterday we obeserved unsual delays in writing to the directory. On 'ls|wc -l' we found >300,000 files in there. Did a quick strace on the process and found it full of mutex calls (More than 3/4 calls in the strace were mutex).

So i thought that new file creation is taking time becasue the system has to every time check certain things (e.g name of files to make sure that the new file with a specific name can be created) amongst 300,000 files and then create a file.

I cleared the directory and the applicaiton resumed to normal service levels.

My questions

1. Was my analysis correct (It seems cuz the app started working fine after a clear down)?
2. More imporatant, how does the kernel work when you try to creat a new file in directory.
3. Can the abnormal number of mutex calls be attributed to the high number of files in the directory?

Many thanks J

Please read about the Linux Filesystem, i-nodes and d-nodes.

http://en.wikipedia.org/wiki/Inode_pointer_structure

The file system is organized into fixed-sized blocks. If your directory is relatively small, it fits in the direct blocks and things are fast. If your directory is not too big, it fits in the direct blocks and some indirect blocks, and is still reasonably fast. If your directory becomes too big, it spills into double indirect blocks and becomes slow.

Actual sizes depend on file system and kernel configuration.

Rule of thumb is to keep the directory under 12 blocks, depending on your block size. Many systems use 8K blocks; a fast directory is under 98,304 bytes.

A file entry is something like 16*4 bytes in size (IIRC), so plan on no more than 1500 files per directory as a practical upper limit.

Directories with large numbers of entries are often slow - how slow depends on the underlying filesystem.

The common solution is to create a hierarchy of directories, so each dir only has a few hundred entries.

Mutex system calls are a result of the application (probably something in the JVM or the Java libraries) making mutex calls.

Synchronisation internal to the kernel you will not see via strace, as this only examines system calls themselves.

A directory with lots of files should not become inefficient if you are using a filesystem which uses directory indexes; most now do (ext3 does optionally but it's normally enabled nowadays).

Non-indexed directories (like those used on the bad old filesystems - ext2, vfat etc) get really bad with lots of files, and you'll see the "open" system call taking a lot longer.