bit-tech.net

Microsoft details Windows 8's ReFS file system

Microsoft details Windows 8's ReFS file system

Microsoft's ReFS promises exascale storage potential and some neat new data protection functionality.

Microsoft has released the first firm details of ReFS, the new file system that will d├ębut with the launch of Windows 8 as a successor to the ageing New Technology File System (NTFS) standard.

In a highly detailed post to the Building Windows 8 MSDN blog, development manager Surendra Verma explains the ins and outs of the Resilient File System, or ReFS, which is designed to update the NTFS format that was first introduced with Windows NT back in 1993 as part of Windows NT 3.1.

Just as NTFS was designed to succeed FAT, ReFS succeeds NTFS. It takes a subset of the features that are widely adopted from NTFS, adding new abilities to protect against corruption and scale to extreme file system sizes.

'The key goals of ReFS,' Verma explains, 'are: Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint; Verify and auto-correct data; Optimize for extreme scale; Never take the file system offline; Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.'

The means of achieving these goals are various. ReFS will include checksums that provide integrity checking for metadata, along with integrity streams designed to provide - optional - protection for user data; a copy-on-write transaction model provides a more robust update mechanism; disk scrubbing is implemented to protect against latent disk errors; and the usual NTFS features including BitLocker encryption, access control lists and symbolic link support make the transition too.

The new file system isn't just about improving data resiliency, however, but about making the march to exa-scale computing. Where NTFS tops out at 16TB in its traditional implementation - though the specification allows for 16 exabytes (EB) in a single file system - Windows 8's implementation of ReFS will support file systems as large as 16EB (16,777,216TB) where possible. If that wasn't enough, the specification itself extends to 256 zettabytes, or 274,877,906,944TB.

While ReFS drops some features from the NTFS specification - including named streams, object IDs, short names, compression, file-level encryption via EFS, user data transactions, hard links, extended attributes and quotas - the standard is likely to prove popular among high-performance computing enthusiasts who need large-scale storage systems with commercial support.

Sadly, there are a few caveats. Unlike the move from FAT to NTFS on the desktop, there will be no way to convert an existing volume to ReFS without creating a new volume and copying the data manually. There's no support for booting from ReFS, with NTFS being retained as the file system of choice for boot drives; and you can't use ReFS on removable drives.

Unlike other large-scale file systems, there's also no deduplication facility or second-level caching facility. 'One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS,' Verma explains. 'ReFS does not explicitly implement a second-level cache, but customers can use third-party solutions for this'

While ReFS is currently a server-oriented file system, it provides a glimpse of the future: after all, NTFS started in Microsoft's server and workstation product line before making its way to the desktop with Windows 2000.

Are you pleased to see Microsoft looking towards the future, or disappointed that certain NTFS features - including the handy filesystem-level compression function - won't be around in future generations? Share your thoughts over in the forums.

33 Comments

Discuss in the forums Reply
bowman 17th January 2012, 13:09 Quote
So they've copied some features from (the open source) ZFS, whilst getting rid of a couple of inferior features ZFS has done better (encryption and compression).

:| A step forward for those chained to the MS system, I s'pose, but not impressive.
l3v1ck 17th January 2012, 13:22 Quote
Quote:
Sadly, there are a few caveats......
Sounds like NTFS will still be the file system of choice for consumers for a long time to come then.
Madness_3d 17th January 2012, 13:54 Quote
  1. There is nothing wrong with NTFS
  2. You can't boot off it, what's the point in it being resilient?
  3. No Hard Links makes life harder for anyone with a small SSD Boot Drive
  4. Only purpose I can see it having is as a backup drive (which can't be "removable")

Grrr Technology is driving me mad atm. If it's not this it's Intel's GPU's "supporting" DX11. Woop de frikin doo, I can now run better effects but at 3fps because I don't have the GPU grunt to use them. :(
Picarro 17th January 2012, 14:14 Quote
So this is useful if I want to create a huge server/database running Windows 8? Why anyone would do that though...
Guinevere 17th January 2012, 14:18 Quote
Quote:
Originally Posted by Picarro
So this is useful if I want to create a huge server/database running Windows 8? Why anyone would do that though...

It's debuting with Windows 8, not Windows 8 only.
Optimaximal 17th January 2012, 14:36 Quote
Quote:
Originally Posted by Picarro
So this is useful if I want to create a huge server/database running Windows 8? Why anyone would do that though...

Server 2008 & 2008 R2 are pretty code-synonymous with Vista and 7. No doubt Windows 8 will share it's base with Server 2012/whatever it's called.
feathers 17th January 2012, 15:04 Quote
Pointless.
ZERO <ibis> 17th January 2012, 15:11 Quote
Interesting I wonder if it will get a boot option at a later date for win 8. Dealing with very large volumes of millions of very small files along with hundreds of very massive files really has had me bumping into the limitations of NTSF for a year now. I am really looking forward to a new file system!
Fantus 17th January 2012, 15:19 Quote
Quote:
Originally Posted by Madness_3d

No Hard Links makes life harder for anyone with a small SSD Boot Drive

Hard links can only be made within a physical drive anyway; Symbolic links can be made across different physical drives and partitions so there's no real loss of functionality.
Hakuren 17th January 2012, 15:36 Quote
Absolutely ludicrous idea. Retaining NTFS for boot purposes and moving everything else to RFS. Whole bloody point of increased security should be booting from new RFS partitions. NTFS is really old and balancing on the edge of uselessness when dealing with really large amount of data. Why they didn't thought about retaining FAT for boot partitions when they introduced NTFS? Because it was utterly moronic idea then. And so is keeping NTFS as a enforced boot partition choice while all other data is running on RFS. Another ble$$ing from Macroco$t.
schmidtbag 17th January 2012, 16:26 Quote
ahem....

phoronix [dot] com/scan.php?page=news_item&px=MTA0NDA

its directly competing to btrfs, not as much of zfs. and even at that, its still missing features.
[-Stash-] 17th January 2012, 16:28 Quote
I welcome this. While it's still not as good as ZFS/BtrFS from the sound of this article, I for one am glad to see Windows get end-to-end checksumming on it's "data fs" - I've already had several GB of photos silently corrupt at some point (on a mechanical HDD) and would really rather it didn't happen again.
GoodBytes 17th January 2012, 16:40 Quote
Microsoft said on the blog site that this file system is for Windows 8 SERVER edition, not home. "Home edition" is coming later on (probably once its bootable, and has features more relevant for home computers).

I know this article didn't make it clear, but that's the case.
Malfrex 17th January 2012, 17:40 Quote
Quote:
Originally Posted by Fantus
Quote:
Originally Posted by Madness_3d

No Hard Links makes life harder for anyone with a small SSD Boot Drive

Hard links can only be made within a physical drive anyway; Symbolic links can be made across different physical drives and partitions so there's no real loss of functionality.

You can make hard links to another partition, however they have to be to a folder, not to a file. They're actually called junctions, but they are still technically hard links. I use them so Windows is happy with it's default folder locations and yet I can have the multiple users on my system have all their pictures/music/videos in the same location but to the computer it looks like its in c:\users\<name>\...
Fantus 17th January 2012, 17:59 Quote
Quote:
Originally Posted by Malfrex
You can make hard links to another partition, however they have to be to a folder, not to a file. They're actually called junctions, but they are still technically hard links. I use them so Windows is happy with it's default folder locations and yet I can have the multiple users on my system have all their pictures/music/videos in the same location but to the computer it looks like its in c:\users\<name>\...

But only on the same physical device. Symbolic links do the same job and are far more versatile.
It doesn't make sense to keep supporting Hard Links when Symbolic links can do the same and more.
Carpet3 17th January 2012, 19:34 Quote
Can't boot? What a stupid design.
GoodBytes 17th January 2012, 19:57 Quote
can't boot YET.
ZERO <ibis> 17th January 2012, 23:35 Quote
I do not understand why they are holding it off maybe add it to win 8 ultimate but still I would love to have all the features early on. Besides it is not as though anyone is going to expect it to work perfectly until they have used it for a year without problems. I am so exited about this I almost want to have a launch party for it! I wish I could just use it for the boot drive and then I can run a test bed system on it for a year and then switch everything over if it looks good. I wish I could have it right now and start testing. Waiting a year or so is just way to long. Lets get off NTSF and move onto the future!!!!!
GoodBytes 17th January 2012, 23:41 Quote
Servers have RAID backups, and separate backup. If anything... data is more than safe.
Most average users does not. Imagine if it fails... data loss, people go on the internet and complain like no tomorrow, turning Win8 into Vista.

Also, based on my understanding is that, in it's current form, it offers no real advantages to the consumer, plus it's not bootable. I would expect it to show on SP1 of Win8, and bootable, improved, set as default, on Win9.
dr-strangelove 18th January 2012, 02:38 Quote
And they said no one would ever need more than 640KB...
rogerrabbits 18th January 2012, 04:24 Quote
274 billion terrabytes :) Just to think... some day... some kid will have a 274 billion terrabyte SSD and will complain that it's too small.
LordPyrinc 18th January 2012, 05:53 Quote
^ Agreed. May take a few years yet for that, but compared to the meager 2GB hardrive powering my first PC in 1996, I don't doubt it.
8w65vw85 18th January 2012, 08:14 Quote
The main thing... For those of you who want this feature in your PCs just because it is new... we have a word to describe this kind of users, we call them NOOBS. Yep, jut because something is new does not mean it is good and even if itis goood, it is not necessarily good for YOU, this kind of snapshooting FSs are not exactly efficient when it comes to say, gaming or video processing. even databases! yes, BTRFS is a bad idea for a DB, even ZFS, which is in my opinion the best FS available for many things is not the best at all when it comes to Databases (random seek + RT + Sync R/RW IO + snapshooting = oh no no no)
Just the same thing happens with Linux desktop users going for BTRFS, it is usually a very very bad idea. Yes, plenty of people preach the marvels of snapshoting, hard/soft-links and hybrid FSs, but in the real world, most people, even those who think themselves 'advanced' will never go read the documentation to understand and know how to use them.

About hardlinks, I have two things to say, 1) Soft links are superior, 2) I dont feel like explaining it, go read the Wiki or something, or believe me blindly (bad).

About the "OMG what kind of useless crap is that sh*t if it does not even boot!"... This too depends on the partition table format and the capabilities of the BIOS to boot the FS and to understand the table. Microsoft could probably just use NTFS compatibility to make ReFS bootable, or adopt a better partitioning table instead of the obsolete msdos format.... maybe Miccrosoft will just adopt the BSD style table!.... hold on we are talking about M$, they never have good ideas of their own and they never work with anything non propietary... and they never work with something that has not been tested only by them and exclusively by them, so the hell with M$ but I hope they open support for other OSes, even make it opensource why now... .... yeah sure, dream on...

and... I was going to say something more but forgot, I havent sleep in two days, actually Im off to bed now....
Bionic-Blob 18th January 2012, 23:04 Quote
Quote:
Originally Posted by Carpet3
Can't boot? What a stupid design.

Doesn't need to be bootable; that's what your 100mB hidden NTFS partition is for (in the default layout).
GoodBytes 18th January 2012, 23:17 Quote
Actually this is Windows repair tools, in the case you lose or don't have with you, the OS disk.
rogerrabbits 19th January 2012, 02:35 Quote
I quite like the idea of a future where my own data no longer destroys itself.
Bionic-Blob 19th January 2012, 03:11 Quote
Quote:
Originally Posted by GoodBytes
Actually this is Windows repair tools, in the case you lose or don't have with you, the OS disk.

No, it is where the BCD bootloader is located, which is why it's the active/system partition.

Windows repair tools are stored in the recovery folder on your OS partition.
GoodBytes 19th January 2012, 03:42 Quote
active/system partition is my System partition (where Windows is installed).
I just loaded a partition disk software, and I don't see such boot partition.

System is Windows including the BCD, and Recovery is auto created by the setup, which has the recovery part of the system.

I don't see an independent partition. I looked on a third computer, which is the most default setup you can possibly ask: clean HDD, no added partition manually. It has 2 partition, Recovery and Windows (has everything like any default config computers).

In fact, if you simply show hidden folder and show hidden system folders, in the folder option panel, you can see on the C:\ drive that you have: BOOTSECT.BAK, bootmgr, and a Boot directory complete with BCD system file.

Hidden partition doesn't offer any security, it just makes "My Computer" location, nicer. Like hiding files and folders. They are still accessible by anything, just hidden. (Well for partition it requires a bit of work, but still possible).
Bionic-Blob 19th January 2012, 04:25 Quote
This explains what I'm talking about:

http://technet.microsoft.com/en-us/magazine/gg441289.aspx

If you can't see it on any of your win7 machines, you most likely installed to a single partition rather than a disk, which sorts out the partition layout automatically.

What I'm trying to get at is that MS have said you cannot boot to REFS, however if we were to have an NTFS system partition with a boot loader that is REFS-aware, we could potentially boot an OS installed on REFS. I can't find any info to confirm this, but it's early days.
GoodBytes 19th January 2012, 04:39 Quote
Ah ok I see.. maybe it doesn't do it, because I have Pro edition of Win7, and not Ultimate.
It's clear that it needs to do this, because how do you decrypt a whole drive to access the OS.
PCBuilderSven 19th January 2012, 19:22 Quote
Quote:
Originally Posted by Madness_3d

[*]There is nothing wrong with NTFS

Other than it needs defragmenting (on a hard disk) and takes up loads of space formating. Switch to EXT4 and make us all happy.
GoodBytes 19th January 2012, 19:28 Quote
Quote:
Originally Posted by PCBuilderSven
Other than it needs defragmenting (on a hard disk) and takes up loads of space formating. Switch to EXT4 and make us all happy.

Every drive format you need to defrag. Basics of how storage works. Even your RAM needs defragmentation... well... on the RAM data isn't split into 2 or more.. it's more having the total free space, but no space to insert anything in 1 block. No mater how you see it it needs defragmenting, which takes time, and loads of space. In the case of your RAM, it's the exact size of the RAM on your HDD.

If you find a way to make a file system that doesn't need any defrag (or just solve the need of data block splitting , you will revolutionize the tech industry. So far, since the early days of computers, no one found any better way the what was specified back in the early UNIX days. All file system is based on that method, with minor tweaks, and added stuff like security (in a nut shell).
8w65vw85 20th January 2012, 09:23 Quote
The boot up restriction is in the BIOS and the boot loader. GRUB can boot anything for which it has a driver (module), Microsoft could just adopt Grub2 and problem solved but they never use anything that is not closed and from them, that is one of the things that I dislike so much about them, like why no to include default support for ogg and flac? well because they want to make money of their own format... if only Microsoft would worry a lot more about technology and a hell lot less about making money (the greed machine)
.

Anyway, there are many excellent FSs that are non bootable, File systems are like scalpels, there are different sizes and they all fit different needs, and of course we have the swiss knives that can do anything and excell at nothing. It is not possible to create a FS that fits all and every need in use today, either for SSD or HDs or other storage devices.

Somebody blames their data vanishes like magic from drive? sorry but that doesn'tt happen, with all honesty NTFS is NOT a bad FS even if ageing. So dont blame the FS or the computer, it is most likely a user fault, or very low end computer parts, I have ran several servers and pcs in my life and have seen an FS corruption only once and it was with ReiserFS, which is actally a much better FS than NTFS.

Also about the defrag, no, not ALL drives require defragmentation, in the case of SSDs it is called relocation, and some modern FSs do NOT need a manual defragmentation since this defrag is made on the fly and in real time.

One more correction, ram is not defragmented, system simply alocate whatever the pointers indicate, and in case something does not fit , there is a cache ram part. ram divides in occupied blocks, cache blocks and free bocks. Your confussion may come from the fact that many applications reserve certain range of blocks, and if no such range is available, some other data cells are relocated, but a defragmentation per se does not exist in ram


Finally, FSs like REFS or BTRFS are animals of a diferent class than ext or ntfs, and different from ZFS and XFS. It is cats vs whales vs robin birds.

Personally, although I welcome an additional FS as long as it is open or at least OS agnostic... I would rather promote the standarization of ZFS in all the three main OSes, it is already on its ways for Linux, and unixes it is already native, it is only mac and MS that would require work but the source code is already available. using standards is a good idea... why MS never follows them unless it is enforced? oh the misteries of life...
Log in

You are not logged in, please login with your forum account below. If you don't already have an account please register to start contributing.



Discuss in the forums