Go Back   IceInSpace > Equipment > Software and Computers

Reply
 
Thread Tools Rate Thread
  #1  
Old 28-04-2017, 11:29 AM
LostInSp_ce's Avatar
LostInSp_ce
Unregistered User

LostInSp_ce is offline
 
Join Date: Mar 2017
Posts: 754
Changing allocation unit size for performance gains

Hi all,
I'm currently tweaking the rig to suit AstroPhotography and wanted to know if anyone bothers with changing their drives from the default values to increase transfer rates and if so do you find the improvement worth while? I talking real world improvements and not bench marks. I was thinking that with the amount of subs that are written/read and considering their file size that maybe increasing the size of allocation units will increase performance during capturing/processing. Thoughts?
Reply With Quote
  #2  
Old 28-04-2017, 08:22 PM
Wavytone
Registered User

Wavytone is offline
 
Join Date: Jul 2008
Location: Killara, Sydney
Posts: 4,147
The greater problem in Windows is disk fragmentation.

With spinning media the ideal is to have a file written to consecutive disk sectors so that as the disk rotates the file is read sector by sector as the disk rotates, with zero waiting in between. When the drive is frganmented, a file is written to one block here, another block there and quite possibly the CU has to wait for the best part of a rotation of the platter to get to the next block in the sequence. On a 7200 rpm drive that could be 140 microseconds waiting between blocks. Repeat for say 1000 blocks and thats a 1.4 second delay which is significant.

However, with spinning rust the read/write time is fundamentally limited by the rotation rate of the platter.

Best thing you can do is eliminate the problem entirely by installing an SSD (no spinning media). The next best is to use a RAID array of multiple drives, though you had better have a robust backup strategy. The zero-cost option is to defragment the drive regularly, but it won't achieve much of an improvement.

Last edited by Wavytone; 28-04-2017 at 09:06 PM.
Reply With Quote
  #3  
Old 28-04-2017, 10:34 PM
LostInSp_ce's Avatar
LostInSp_ce
Unregistered User

LostInSp_ce is offline
 
Join Date: Mar 2017
Posts: 754
Thanks Wavy, the main drive running OS/software is a SSD and this is a laptop not my main system so there's no raid config. The drive I want to increase the block size on is an external USB 3.0 that I'd use away from home.

I thought that by doing this I could capture directly to the larger external drive rather than to the smaller SSD which would eliminate having to transfer hundreds of subs to the external drive at the end of the session. I understand that read/write speeds are slower with USB but thought that maybe I could get some of it back by using bigger blocks. Also I wouldn't be storing any small files only subs so there shouldn't be too much drive space wasted using a 64-128kb block size.

Is this worth doing or would the gains just be too little to even bother with?

Update: After a little research and looking at my sub sizes I've decided to try out exFAT at 1024kb.
Update: Upped it again today now using 4096kb and seems to performing well so far.

Last edited by LostInSp_ce; 30-04-2017 at 11:44 AM.
Reply With Quote
  #4  
Old 02-05-2017, 10:28 AM
sil's Avatar
sil (Steve)
Not even a speck of dust

sil is offline
 
Join Date: Jun 2012
Location: Canberra
Posts: 1,474
1) Windows and USB drives ALWAYS ALWAYS ALWAYS disconnect them from the tray icon and only unplug after you get the message saying its ok to do so. You will lose data and corrupt the drive, you've been warned.

2) I'd make you have no problems with reading the data back on another machine/software as part of the testing. So assuming you capture on a laptop make sure your desktop has no problems with the drive and files.

3) make sure you dont run a realtime virus scanner, it will interfere with writing the video files to the drive.

4) Avoid HDD and use only SSD to capture, ideally laptop boot drive should be SSD too just so windows spends less time interfering with IO.

Back in the day I used to do 3D stuff for TV and film and sometimes got to use proper Digital Video Recorders. These were standalone units with one or more hard drives built in and could take a video input or serial trigger still frame capture (to build a movie frame by frame). These weren't usb devices, they were standalone and would interface with whatever video equipment you have for display on location or conversion to your master format later. Bulky but far more compact than a portable Betacam SP setup. No doubt the same sort of thing is pocket size these days and able to record uncompressed from a usb camera.

Just thinking though what is the problem with leaving things as they are? What are you trying to improve? Whats your capture gear?
Reply With Quote
  #5  
Old 03-05-2017, 03:16 AM
LostInSp_ce's Avatar
LostInSp_ce
Unregistered User

LostInSp_ce is offline
 
Join Date: Mar 2017
Posts: 754
Thanks for the heads up Sil. There's nothing wrong with how things are but I had a spare new drive just sitting here and thought that I should use it with the laptop to increase storage capacity as well as for transferring subs to back to my main system or another computer if I'm not on the network. I could also leave them on there making it a backup.

Then I thought that I should just save the subs directly to the external USB drive to save some time from having to transfer them all at the end of each imaging session.

Since the drive will only be used to store subs I figured why not optimise it to suit the size of my subs. Being USB I know that read/write speeds will be slower than the internal SSD but thought that with an increased block size I might be able maximise what's left.

I understand you will always be limited by the speed of the USB interface but data speeds never reach their theoretical potential. Anyway I've been playing with different block sizes in relation to my sub sizes and found that using 4096KB has made a difference. Read/write speeds are faster than the default 128KB. It's nothing revolutionary but it is an improvement and it didn't cost anything to try.

I'll probably get a bigger SSD later but for now this seems to working well.
Reply With Quote
  #6  
Old 04-05-2017, 02:39 PM
sil's Avatar
sil (Steve)
Not even a speck of dust

sil is offline
 
Join Date: Jun 2012
Location: Canberra
Posts: 1,474
block size will mostly effect fragmentation and possibly write speeds. I doubt theres much to be gained for you and more to lose if your drive developes errors and you need to salvage data you may have problems reading weird block sizes. I agree it costs nothing for you to have a play. If you're storing lots of small sized files it may be a benefit. it may be better for you to look into a small NAS unit that can use a raid format to get performance and/or data safety. Ultimately though AP take a lot of data and we need it uncompressed to keep signal as accurate as possible and by its nature we're stuck with tons of large filesizes. So the larger block sizes can make sense though you may not be making best use of the drive capacity if you have smaller files on the drive as well as huge videos. it all depends on how many blocks each file takes up and the remainders still take up blocks even if they dont use the space.
Reply With Quote
  #7  
Old 04-05-2017, 04:05 PM
LostInSp_ce's Avatar
LostInSp_ce
Unregistered User

LostInSp_ce is offline
 
Join Date: Mar 2017
Posts: 754
Thanks for outlining some of the potential risks. It will only store subs so there really shouldn't be any other file types apart from the odd hidden system file written by the OS I guess. You're right about using a NAS unit and if this was the main rig then this would definitely be something to consider. However the laptop will only be used with the external drive every now and then at a dark site or weekend getaway. So far as a cheap solution it seems to be working out OK and from a worst case scenario if I was to lose anything it would probably only be a night or two of imaging. Although annoying I don't see it as the end of the world.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 11:11 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Testar
Advertisement
Bintel
Advertisement