quickly creating / reserving space for large files, not dd?

Code junkies hangout here

Moderators: ChrisThornett, LXF moderators

quickly creating / reserving space for large files, not dd?

Postby silly_xp_user » Sat Dec 30, 2006 6:55 pm

Hello,

In winxp, with an NTFS disk, you can issue the command fsutil file createnew big.bin 123456789012 to create a large zeroed file and this works almost instantaneously because it does not write data to disk, it uses some win32 api calls to reserve space. This is not the same as a truly sparse file since the size on disk is the correct size.

I wondered if there was a linux equivalent. The traditional dd if=/dev/zero ... will obviously take time.

If there is no obvious way then are there other methods of quickly reserving space?

Thanks for any help given.

Hal
User avatar
silly_xp_user
 
Posts: 11
Joined: Sat Oct 22, 2005 3:55 pm

RE: quickly creating / reserving space for large files, not

Postby drws » Sat Dec 30, 2006 7:14 pm

you can still use dd to create the file just do
Code: Select all
dd if=/dev/zero of=filename bs=1M count=1 seek=50

that should create a 51Mb file by writing 1Mb of data but seeking 50meg into a file first.
ls should show the file as 51Mb, but du should show its true size (1Mb)

I set the block size to 1Mb just to make the maths easier (the seek is how many multiples of block size to skip in the output file)
You could make it bs=1 for 1 byte and then set seek accordingly.
drws
LXF regular
 
Posts: 125
Joined: Sat Aug 06, 2005 10:39 am

RE: quickly creating / reserving space for large files, not

Postby silly_xp_user » Sat Dec 30, 2006 7:51 pm

That's a good start but in your example, I would want it's true size to be 51Mb and would want du to show it as 51Mb.

Hal
User avatar
silly_xp_user
 
Posts: 11
Joined: Sat Oct 22, 2005 3:55 pm

RE: quickly creating / reserving space for large files, not

Postby MartyBartfast » Sat Dec 30, 2006 8:43 pm

Well how about a slight tweak on drws' solution:

~$ dd if=/dev/zero of=a.a bs=1k count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.005958 seconds, 172 MB/s
~$ ls -l a.a
-rw-r--r-- 1 martyn martyn 1024000 2006-12-30 20:39 a.a
~$ du -b a.a
1024000 a.a
martyn@inspiron:~$
I have been touched by his noodly appendage.
User avatar
MartyBartfast
LXF regular
 
Posts: 814
Joined: Mon Aug 22, 2005 7:25 am
Location: Hants, UK

RE: quickly creating / reserving space for large files, not

Postby silly_xp_user » Sat Dec 30, 2006 8:56 pm

That was plan A but as my initial post said: "The traditional dd if=/dev/zero ... will obviously take time".

The key thing is to reserve the space and not spend time writing to the file. The first example I gave reserved space for a 100GB+ file.
User avatar
silly_xp_user
 
Posts: 11
Joined: Sat Oct 22, 2005 3:55 pm

RE: quickly creating / reserving space for large files, not

Postby drws » Sat Dec 30, 2006 10:29 pm

Windows seems to create a sparse looking file, in that it is created quickly without major disk writes and contains all zeroes when inspected with a hex editor.
Yet, as mentioned, its occupies whatever disk space you ask for meaning it isn't a sparse file.
I cannot create a file bigger than my available free disk space. (Which you can do using the 'dd' trick in linux. - because this is a sparse file)

At first I thought that NTFS was just updating its allocation table to say 'these blocks are in this file', but then I would expect the file to contain data (whatever was on that section of the hdd previously)

What can we call these sparse-looking zero-filled disk-consuming non-sparse files that NTFS creates? :)

Apart from manually writing to the disk, I can't think of a way to create a real disk-occupying file quickly.
Unless you want to write 1 byte per cluster instead of the full 4k (assuming default 4k cluster size) - would this be quicker?
Anyone else got any ideas?
drws
LXF regular
 
Posts: 125
Joined: Sat Aug 06, 2005 10:39 am

RE: quickly creating / reserving space for large files, not

Postby drws » Sat Dec 30, 2006 10:47 pm

Another thought
Do you want to create a file, or just reserve disk space?
You could reserve more space for root on a partition using tune2fs. It is usually set to 5%, so non-root users see the disk full at 95% instead of 100% etc.
drws
LXF regular
 
Posts: 125
Joined: Sat Aug 06, 2005 10:39 am

Re: RE: quickly creating / reserving space for large files,

Postby silly_xp_user » Sun Dec 31, 2006 11:01 am

At first I thought that NTFS was just updating its allocation table to say 'these blocks are in this file'

Even that would be fine - how would you do it in linux?
Unless you want to write 1 byte per cluster instead of the full 4k (assuming default 4k cluster size) - would this be quicker?

Dunno but if we scrap the requirement that it's a (completely) zereoed file then how do you do this?

Perhaps the original request query of creating a 'reserved space pseudo sparse' file might might be one for a C program? Any takers?

I may have to resort to tune2fs but would would prefer a simple file creation tool/script/program.

Hal
User avatar
silly_xp_user
 
Posts: 11
Joined: Sat Oct 22, 2005 3:55 pm

RE: Re: RE: quickly creating / reserving space for large fil

Postby silly_xp_user » Sun Dec 31, 2006 11:06 am

What about creating a large swap file and not using it?
Is this fast or do I still need to use dd to fill/size it?
Hal
User avatar
silly_xp_user
 
Posts: 11
Joined: Sat Oct 22, 2005 3:55 pm

RE: Re: RE: quickly creating / reserving space for large fil

Postby drws » Sun Dec 31, 2006 2:49 pm

i tried creating a sparse file and then formatting for swap usage, and although it looks like a large swap file, its still only occupying a small amount of disk space. ( also tried using mkfs as well)
I've been looking a how programs that reserve disk space work and they either create the sparse file, or take ages creating a blank file.
I would have thought if there was an obvious way then some of those developers would have used it.
drws
LXF regular
 
Posts: 125
Joined: Sat Aug 06, 2005 10:39 am


Return to Programming

Who is online

Users browsing this forum: No registered users and 2 guests