2

The problem was already discussed here. But there was not consensus on this topic.

I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-based structure (eg ext4, ntfs, probably btrfs) we can utilize this feature to make modification of middle file's parts independent of other its parts. I suppose it will require handling of length for each such part independently of other's. But in some situations advantage may be drastically big. From my experience: I sometimes faced with the problem of slow processing of one big file. So the functionality may be in demand. And I don't even mention databases here which always required such funcionality.

A good usage case would be distribution of read/write operations across multiple disk layers. It is rather relevant for modern multidisk (often ssd-based) multicore and mulithreaded SMP (or even NUMA) systems.

I already took a look at MPI-IO v.2 system. It has something simialar (especially regarding parallel processing). But it does not provide dynamic file resizing capability which I suppose to introduce.

I need your opinions on this topic. What drawbacks / shortcomings can arise in trying to implement such feature. One of such drawback can be odd irregular length of contigous file blocks resulting in breaking memory mapping mechanisms for example. I just want to notice that someday such functionality will be implemented because:

  1. Data volumes grow rapidly and so files also do
  2. Parallel processing is already a modern reality. And there are still no other good ways of future technology improvement in terms of performance.
Anton Astafiev
  • 64
  • 6
  • There *is* consensus on that other post. It states that *file systems are composed of blocks,* and as long as they are, you will have to deal with the limitations so imposed. Create a file system where every byte is individually addressable directly, and you won't have this problem anymore. – Robert Harvey Jan 14 '20 at 15:56
  • 1
    Of course, there are reasons why file systems were composed of blocks in the first place, and you would have to give up those advantages to create your new file system. Speed is one of them; require addressing each byte individually and you will take a tremendous performance hit. It will also cost you dearly in terms of storage efficiency. Is it worth it, considering that the workarounds for ordinary block-style file systems are trivial and well-understood? – Robert Harvey Jan 14 '20 at 15:58
  • 1
    So yes, it is possible. Anything is possible given enough time, money and donuts. Is it practical? Probably not. – Robert Harvey Jan 14 '20 at 16:02
  • 1
    @RobertHarvey, yes of couse I took into account all opinions presnt in linked question. But there were almost no alternatives to the mainstream fs design present. And I am sure that main obstacles are not the granularity of I/O acesses (one can and should continue to operate with independent 4k block I'm sure) – Anton Astafiev Jan 14 '20 at 16:06
  • 1
    To find out what drawbacks will arise when you try to implement a feature, you could try to implement a feature. – user253751 Jan 14 '20 at 16:41
  • 1
    Databases just roll their own insert mechanism. The block nature of the devices doesn't stop them from managing inserts, which they can each do with their own custom application-specific approach, probably way better for them than having insert in the file system. – Erik Eidt Jan 14 '20 at 17:02
  • Databases can manage files the same way other applications manage memory: as a large linear array, to be carved into blocks, some of which are in use and some are free. As such the block layer isn't that hard to work with. – Erik Eidt Jan 14 '20 at 17:03
  • Some databases use memory mapped files, which gives them byte/word access, and use paging for I/O -- though a high performance database will implement custom application-specific paging. – Erik Eidt Jan 14 '20 at 17:03
  • @ErikEidt, yes AFAIK DBs implemented their own filesystems with insertion capability available – Anton Astafiev Jan 14 '20 at 17:03
  • @ErikEidt, "way better for them than having insert" - this also can mean funcionality dublication – Anton Astafiev Jan 14 '20 at 17:07
  • Yes, but basically, you want to push this functionality up the stack so that individual applications can take advantage of their domain-specific knowledge, rather than pushing it down the stack going for one-size fits all. – Erik Eidt Jan 14 '20 at 17:20
  • @AntonAstafiev: Oracle for instance does all I/O in block sizes. All the modifications happen in memory, and are flushed out in blocks. If a row grows, the whole block is re-written (in memory) before being flushed out. There are marginal cases when rows span multiple blocks or are 'migrated' from one block to another (needing intermediate "pointers") - guess what: those are performance killers. – Mat Jan 14 '20 at 19:18
  • @Mat, this sounds almost like NTFS directory's or BTRFS B-trees – Anton Astafiev Jan 14 '20 at 19:46

0 Answers0