linux - How to make the OS schedule disk accesses optimally? -

Suppose a process requires access to file systems in many systems (1000+) locations, and for order Not important is the programmable logic, however, the order is clearly meant for performance if the file system is stored on (spinning) hard disk.

How can the application programmer communicate the OS that it should optimize access? Launching 1000+ threads does not seem practical. Does the database management software accomplish this, and if so, how?

Additional details : I had a large (1TB +) mmified file where I needed to read about 1000K amounts of 1KB For, every time, in new, unexpected places.

Rearrange the read / write commands already in the kernel (like the mechanical disk spin To fit), if they come from different processes or thread BTW, mostly reads & amp; Writes, not on the disk, the kernel files will go into the system cache.

You can consider using & amp; Perhaps (in separate threads) if you use some file part to project in virtual memory, you can also use it

Of course, the file system generally does not guarantee this That a sequential part of a file is sequentially located on disk (and even the disk firmware sector can be re-ordered) see the picture in the wikipage, also relevant to it. Some file systems may be better in that regard, and you can tune your block size (at mkfs timings).

I do not recommend having thousands of threads (only a few dozen).

Finally, it may be possible to buy some or more RAM (file cache)

The actual display will depend greatly on special systems and hardware.

Perhaps it is appropriate to use an indexed file library or database library (or a real database such as PostGreSQL)! There are probably fewer files but big people can help.

BTW, you are mmap - reading, and reading a small portion of 1 (less than 4K's page size). You can use the madvise (if possible in advance), but you should try to read large amounts, because using every file will bring at least one whole page.

You need benchmark!