Jump to content


Welcome to DBSTalk


Sign In 

Create Account
Welcome to DBSTalk, like most online communities you must register to view or post in our community, but don't worry this is a simple free process that requires minimal information for you to signup. Be a part of DBSTalk by signing in or creating an account.
  • Start new topics and reply to others
  • Subscribe to topics and forums to get email updates
  • Get your own profile page and make new friends
  • Send personal messages to other members.
 
Guest Message by DevFuse

Photo
- - - - -

Transferring individual recordings between eSATA drives - solved!


  • Please log in to reply
24 replies to this topic

#1 OFFLINE   cigar95

cigar95

    Legend

  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 12 August 2011 - 06:20 PM

((ETA - check post #10 to see some information on certain drives on which this process may not produce a watchable recording.))

One of the truly noteworthy threads on the board details how to duplicate the contents of one DVR onto another:

http://www.dbstalk.c...ad.php?t=148760

However, if you have followed some of the threads here and on the DVR board over the last ten weeks, you may remember that I found myself in a situation in which my drive had suffered some unknown file corruption which prevented it from booting the DVR, and the process described in that thread produced a duplicate drive that inherited the same unbootable behavior. Nonetheless, in mounting my drive in Linux, I found that the file system was intact, and the recordings appeared to be present and accounted for. Surely, I thought, there ought to be a way to move them to a fresh drive.

This led to a long discussion that started here:
http://www.dbstalk.c...ad.php?t=193801

and later migrated here:
http://www.dbstalk.c...277#post2811277

With lots of experimentation, trial, and error, I discovered how to transfer *individual recordings* from one drive to another. This could be useful for anyone with a little patience and who isn't afraid of issuing some simple unix commands. If, for instance, all of your recordings of one show are on one drive, and somehow you end up out of town and recording the latest episode to a different drive. Or maybe you want to archive some set of recordings to another drive. This process would work for that as well. You could even use it as a backup process, although the speed of transfer may work against that, vis-à-vis simply copying an entire drive.

Key point - this process will allow recordings to be transferred to a drive that already contains recordings without disturbing the existing recordings. (The same is true of the process for copying the entire drive.) That is critical.

Caveat - this does not defeat the restriction that recordings can only be played on the DVR on which they are originally recorded. Nor is it intended to do so - if you want to use this process to experiment with that, don't mention my name.

Caveat 2 - This requires being able to mount both the source and the target drive in Linux, which means an internal drive must be removed from your DVR. This again returns us to the issue of owned versus leased DVRs, which I'm not going to address - we all know that debate. In my case, the problem child drive was an external, so the issue didn't come up.

Caveat 3 - I have not figured out how to, or if it is even possible to, transfer scheduled recordings or series recording links between machines. Just recordings that have already been made. Maybe someone cleverer than I can do some research into that one. (P Smith, you game?)

ETA: Caveat 4 - This process seems to fail if the target drive is a Fantom Green Drive. This may be related to that drives non-use of the new Advanced format technology, but that's just a guess at this time. See post #10 for a longer discussion.

So, enough blabbering. How do you do this?

Stage 1 - Connect both drives to a Unix/Linux system and mount the filesystems.

This part is identical to the description for duplicating an entire drive, although I will name my drives differently. (The naming is not crucial, just helpful to me.) It assumes that both drives already have a DirecTV system on them. I was able to make this work even though the system on my source drive was slightly corrupted. Obviously, any corruption has to be minor enough that the file system itself and the recordings and their metadata are still intact.

It also assumes that both drives were "gracefully shut down" when last booted in your DVR. Otherwise, you'll have to perform an xfs_repair before they'll mount in Linux. (Search the forum for the invocation syntax of xfs_repair that works in this context.)

The drives can be connected via eSATA or USB - although the latter is about half as fast, and I experienced some reliability issues when copying files. I have used GPartEd Live 0.4.6-1, which works just fine. Although other versions of Linux are known to not work for the process of copying an entire drive, I’m tempted to see if my process will work in Ubuntu or some richer version than GPartEd.

Once you boot, do the following from the Linux command prompt to mount the file systems:

mkdir /mnt/new
mkdir /mnt/old
mount -t xfs -o rtdev=/dev/sda3 /dev/sda2 /mnt/new
mount -t xfs -o rtdev=/dev/sdb3 /dev/sdb2 /mnt/old

Your device designations may be something other than "sda" and "sdb" - GPartEd will show you which is which. And be sure that you have the "old" and "new" designations right - if you have them reversed, you'll end up transfering recordings in the wrong direction.

Once you have things mounted, do the following:

cd /mnt/old/viewer/indexfile
ls -1  (last character is a "one", not the letter between k and m in the alphabet)

You will see a full list of your recordings. If it's a long list, you may wish to pipe the output through the "more" filter. (A basic familiarity with Unix will be helpful, but you can get by without it.) Thanks to the DTV engineers, the directory names you see in this list give you a lot of information to help you identify your recordings. For instance, a typical name is:

Rcrd-05-17-2011-2000-00-2-ch4-min65535-src2.mpg

The prefix is always the same. The next part is clearly the date, followed by the start time (using a 24 hour clock) in the form hhmm-ss. The next digit I don't know what it's for. Usually it’s a small number, but a few times I’ve seen it up in the hundreds. The next part is the channel number, and then the rest is always the same. Everything up through the "src2" I will refer to as the "program identifier", since the ".mpg" actually changes in one place.

At this point, to save myself a lot of typing, I'll use "xyzzy" to represent the program identifier, even though the actual names are much longer.

Now to transfer the recording, there are three bits of data that need to be transferred. Two are metadata of some sort, and the third is the actual recorded content. If you look closely at the folder /mnt/old/viewer/indexfile, you will see that the file "xyzzy.mpg" is not actually a file at all, but a directory that contains four files of metadata.

Similarly, the file /mnt/old/viewer/segments/xyzzy.mpg is also a directory - a very *large* directory that contains a series of 16 Mb files. These files are the actual recorded content, and they are stored on the real-time section of the file system. (As far as I can tell, no other data is stored on that partition.)

And finally, another bit of metadata lives in /mnt/old/mw_data/xyzzy.umd.dmd. That file stores the information on where you last left off when playing it, and I believe it stores the location of any bookmarks you have created in the recording.

So we need to move these three chunks of data to the target system, which lives in /mnt/new. The only catch to this is that the data in the "segments" directory has to be copied using the command "xfs_rtcp", since a regular invocation of the Unix command "cp" will copy it to the non-realtime portion of the file system. This portion is "only" 16 Gb, and it would fill up very quickly.

So, the commands to make the transfer are these:

cd /mnt
cp old/mw_data/xyzzy.umd.dmd new/mw_data
cp -R old/viewer/indexfile/xyzzy-min65535-src2.mpg new/viewer/indexfile
mkdir new/viewer/segments/xyzzy-min65535-src2.mpg
chmod a+rwx new/viewer/segments/xyzzy-min65535-src2.mpg
xfs_rtcp old/viewer/segments/xyzzy-min65535-src2.mpg/* new/viewer/segments/xyzzy-min65535-src2.mpg
chmod a+x new/viewer/segments/xyzzy-min65535-src2.mpg

One thing I discovered by accident is that if you do all but the last two steps, you’ve transferred all the metadata, as well as created the folder where the recording data will live. This can be done very quickly, since it’s only the xfs_rtcp command tat takes a long time. AS a result, if you boot the disk on your DVR, these recordings will appear in your list, although, obviously, you won’t be able to play them. This was useful to me because about a quarter of the recordings on my drive I couldn’t know what they were just from their file names. (The others, base on the date, time, and channel of the recording, I was able to figure out.)

You can also see that there is a lot of repetitive typing here. I've found that a moderate knowledge of unix commands is very helpful. The two things that were really useful were the auto-complete command, that will allow you to complete a path and file name by typing just a few characters at a time, as well as the history function, that allows you on the command line to call up previous commands and edit them to invoke new commands.

Also, because I had to copy 163 different recordings, creating a simple Unix script made life a lot easier.

This process is not something that a lot of users are going to need. But I'd never come across someone figuring out how to do this, and so now it's out there for anyone to use. If anyone tries it and discovered a typo in my commands as entered here, let me know. Unless you type something really wrong, all that's likely to happen is that Unix will complain that it can't find a file or a directory. Unless you have a recording on the destination drive with the exact same name (which I don't think is possible), I don't think this will break anything if you have a slight typing error.

Finally, be aware that the real-time copy command is, ironically, very slow. With my eSATA mount, it transferred about 5.5 Gb per hour. That means that a one-hour program in 1080i transfers in about 40-50 minutes. Recordings in 720p, or in standard def, will go correspondingly faster. It was also interesting that recordings of the same length and in the same resolution can take significantly different amounts of disk space - almost a factor of two, at the extremes.

Edited by cigar95, 19 September 2011 - 11:51 AM.
Discovered now information that may result in problems


...Ads Help To Support This SIte...

#2 OFFLINE   P Smith

P Smith

    Mr. FixAnything

  • Registered
  • 19,324 posts
  • LocationBay Area
Joined: Jul 25, 2002

Posted 12 August 2011 - 07:12 PM

Thank you cigar95, soon ppl will try your method and we will have fun ... a lot of fun. :)

Without it it would be boring. ;)

Edited by P Smith, 12 August 2011 - 07:32 PM.


#3 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 16 August 2011 - 01:02 PM

One other thing that I learned from this process that can potentially save some disk space.

Suppose I record a sports event, and I pad it with an extra hour or so in case of overtime. Then the game ends on time and I have an hour of carrot cake recipes on the end of my golf tournament.

Well, if the drive is mounted in Linux, I can go into the "segments" folder for the recording in question, and just delete all the 16Mb recording files that aren't part of the recording that I want. The key is to look at the time stamps on those files to see which ones fall after the time that I want to cut it off

Similarly, let's suppose that I recorded an entire event, and then all I want to save is the last five minutes. I can go into the segments folder and delete those recording files before the part that I want to keep. The recording will still play - it will just start playing at the first file that is still in the folder.

One caveat is that none of this changes the metadata for the recording, so the DVR still thinks that the five hour recording is a five hour recording. It just plays the content that it has. I've long thought that a nice feature of the user interface would be a "trim" function, so that the DVR could intelligently lop off parts of the beginning and end of the recording. And now that I understand how the files are stored, I see that this is not difficult to do.

Another caveat that I have to make clear - if a recording has been transferred from one drive to another, the time stamps on all the transferred files will be the times they were written to the new drive, not the time stamp on the original. So you can't use that to determine which files to delete, you'll have to use your best judgment. (Unless you still have access to the recording on the original drive where they were recorded.)

Whenever you're trying this, my recommendation is not to *delete* the "segment" files you no longer want, but move them to a holding folder of some sort, and store this folder somewhere outside of the "viewer" folder. That way, you can put segments back in if you accidentally removed too many. Once the recording is trimmed the way you want it, you can then delete the folder holding the other recording segments and reclaim your disk space.

Edited by cigar95, 17 August 2011 - 09:44 AM.


#4 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 17 August 2011 - 06:32 AM

P Smith and I have been working on this for weeks. Of course he's a customer.

All this process does is moves recordings from one working DTV setup to another. It can't, as far as I know, be used to pirate content in any way.

(Complaint that this was a response to, apparently deleted.)

Edited by cigar95, 17 August 2011 - 09:45 AM.


#5 OFFLINE   mocarob

mocarob

    Legend

  • Registered
  • 186 posts
Joined: Jul 26, 2007

Posted 14 September 2011 - 01:18 AM

nevermind all that i wrote. (deleted my question)
I know you said to use gparted but as a test, I was using knoppix to see if i could view it in a gui interface and didnt notice that it had mounted in the first try. The dvr drive i had connected was in location sdc. I was the able to open sdc2. (it said that sdc3 was already mounted or busy)

A list of recordings as you described didnt show in knoppix. I was able to open the directory sdc2 and it had alot of folders that werent very big. Other folders were locked so i couldnt drill down to see what was inside. Some of the folder names I could open were named - backup, dms_data, druid_data.
Some of the ones i couldnt were - lvg_data, ssh.

The reason i tried it this way was to see if i could somehow click and drag the files to copy instead of typing commands.
As you can tell, I dont know much about this stuff but wanted to play around anyway..

Edited by mocarob, 14 September 2011 - 01:54 AM.


#6 OFFLINE   mocarob

mocarob

    Legend

  • Registered
  • 186 posts
Joined: Jul 26, 2007

Posted 14 September 2011 - 01:48 AM

Do you think that this line from above could be that some recordings are in
mpg2 vs mpg4 ?

"recordings of the same length and in the same resolution can take significantly different amounts of disk space - almost a factor of two, at the extreme"

#7 OFFLINE   CCarncross

CCarncross

    Hall Of Fame

  • Registered
  • 7,037 posts
  • LocationJackson
Joined: Jul 19, 2005

Posted 14 September 2011 - 01:13 PM

Do you think that this line from above could be that some recordings are in
mpg2 vs mpg4 ?

"recordings of the same length and in the same resolution can take significantly different amounts of disk space - almost a factor of two, at the extreme"


Yes, and the fact that curling probably takes much less bandwidth per hour than football due to the fewer changing pixels. More action and movement means larger file sizes.

#8 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 14 September 2011 - 04:23 PM

Yes, and the fact that curling probably takes much less bandwidth per hour than football due to the fewer changing pixels. More action and movement means larger file sizes.


Curiously, what I have found is that there are sometimes even big differences between different episodes of the same show. In my case, it isn't a difference between mpg2 and mpg4 - I haven't made any HD recordings from the older one in a long time.

One of the biggest "storage per minute" shows I saw just happened to be an episode of "The Biggest Loser" - so it's hard to predict ahead of time what shows will eat up the disk space.

#9 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 14 September 2011 - 04:34 PM

nevermind all that i wrote. (deleted my question)
I know you said to use gparted but as a test, I was using knoppix to see if i could view it in a gui interface and didnt notice that it had mounted in the first try. The dvr drive i had connected was in location sdc. I was the able to open sdc2. (it said that sdc3 was already mounted or busy)

A list of recordings as you described didnt show in knoppix. I was able to open the directory sdc2 and it had alot of folders that werent very big. Other folders were locked so i couldnt drill down to see what was inside. Some of the folder names I could open were named - backup, dms_data, druid_data.
Some of the ones i couldnt were - lvg_data, ssh.

The reason i tried it this way was to see if i could somehow click and drag the files to copy instead of typing commands.
As you can tell, I dont know much about this stuff but wanted to play around anyway..


In order to look at all the data, you need to get the different segments of the xfs volume mounted - it needs to have sdc3 linked as the real-time partition of the sdc volume. You may have to use the "mount" command, or some utility that knows how to mount xfs volumes with real-time components.

While there are lots of folders on there that you can explore, the only ones you really need to look in are viewer/indexfile and viewer/segments. Either one of those will give you the list of programs I described. And then there is important metadata for each recording stored in mw_data.

I have a hunch that unless you have a really smart utility, drag and drop is going to fail because of the real-time partition of the volume, which is where almost all of the disk space is taken up. The default is to copy to the regular partition (sdc2), which is far too small to hold more than a few programs. That's why using cp, which is much faster than xfs_rtcp, fails.

Keep experimenting - just don't do anything that's going to delete data or otherwise trash your drive. There are some things, such as deleting the extra segments to trim the ends off recordings, that could be done in a gui setting, provided that the volume can be mounted properly.

Edited by cigar95, 14 September 2011 - 05:09 PM.


#10 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 19 September 2011 - 11:48 AM

After being very pleased with myself for figuring out how to do the copy, I now have to offer the caveat that it looks like the process doesn't work properly on some drives.

When I did my original testing a couple months back, both my source and my target drives were the Western Digital drives that have been long cited as the best choices for working eSATA drives.

In the meantime, one of my working external drives was a Fantom Green Drive - one that I bought to use with my computer, and just for fun tested it out on my HR-23. When I discovered surprisingly that it worked, I kept using it, mostly for sports events and other things with long programs. My TV series usually went on my WD drive.

After I got the transfer process working, originally for recovering programs from a drive that had gone bad, I used it a few times simply to transfer a couple programs to the Fantom drive because it made more sense for them to sit on that drive with similar programs. I just assumed everything would be fine.

Well, everything was not fine. The transfer process itself went all right, but last week I found that the program would only play for a couple of seconds before coming up "do you want to delete this program?"

So what's wrong with this other drive? Well, I think it might be related to whether or not it uses the Advanced Format technology that the newer WD drives use. I say this because even though all of the transferred segment files are still of size 16777216 bytes, their disk usage is split between 16384k and 16388k, which makes me think there is some sector straddling going on that isn't properly accounted for.

On the WD drives, almost without exception, the segment files take up 16384k after transfer. This is just speculation, but it seems reasonable. Although I think of the Fantom drives as a step down in overall quality from those from WD, mine has worked just fine with my DVR.

I haven't looked into it, but I'm told there is software that can realign a volume on an older drive without the Advanced Format technology to avoid the sector straddling issue. Sounds like doing this may wipe out the drive's content, though, so I'd have to do a backup of some sort. But if there's a way around the problem, that would be great.

Also, I should add that transferring recordings *from* the Fantom drive *to* a WD drive seems to work just fine. It seems to be just the one direction that's problematic.

#11 OFFLINE   ntrance

ntrance

    Legend

  • Registered
  • 145 posts
Joined: Aug 18, 2006

Posted 20 September 2011 - 12:13 AM

You may find that inside the Fantom enclosure, you have a Western Digital drive. I know they have used WD drives in the past. I think can identify the model using hdparm -I /dev/sd? Or you can use another software tool or open the case.

#12 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 20 September 2011 - 06:29 AM

You can get the make and model with GPartEd, as well. Mine is, I believe, a Hitachi.

In the case of a Fantom, no way would I ever open the case - it's a very good bet that I'd never get it properly reassembled.

#13 OFFLINE   mocarob

mocarob

    Legend

  • Registered
  • 186 posts
Joined: Jul 26, 2007

Posted 20 September 2011 - 03:38 PM

In order to look at all the data, you need to get the different segments of the xfs volume mounted - it needs to have sdc3 linked as the real-time partition of the sdc volume. You may have to use the "mount" command, or some utility that knows how to mount xfs volumes with real-time components.

While there are lots of folders on there that you can explore, the only ones you really need to look in are viewer/indexfile and viewer/segments. Either one of those will give you the list of programs I described. And then there is important metadata for each recording stored in mw_data.

I have a hunch that unless you have a really smart utility, drag and drop is going to fail because of the real-time partition of the volume, which is where almost all of the disk space is taken up. The default is to copy to the regular partition (sdc2), which is far too small to hold more than a few programs. That's why using cp, which is much faster than xfs_rtcp, fails.

Keep experimenting - just don't do anything that's going to delete data or otherwise trash your drive. There are some things, such as deleting the extra segments to trim the ends off recordings, that could be done in a gui setting, provided that the volume can be mounted properly.


my plan was to try a version of linux that included gparted. debian w/gnome.. Or ubuntu 10.04.1 has a version of gparted version 5 included.
Since you use gparted for mounting, I thought it might be possible to mount using the terminal commands then see the file system within the ubuntu gui..

Unfortunately I havent had the time to test and I have a corrupt drive i'm using right now so i havent been able to get very far.. Do you think my theory has any chance of working??

#14 OFFLINE   P Smith

P Smith

    Mr. FixAnything

  • Registered
  • 19,324 posts
  • LocationBay Area
Joined: Jul 25, 2002

Posted 20 September 2011 - 03:48 PM

You should finish fixing the XFS file system on Linux machine. I prefer full system like Ubuntu or Fedora.

#15 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 20 September 2011 - 06:06 PM

my plan was to try a version of linux that included gparted. debian w/gnome.. Or ubuntu 10.04.1 has a version of gparted version 5 included.
Since you use gparted for mounting, I thought it might be possible to mount using the terminal commands then see the file system within the ubuntu gui..

Unfortunately I havent had the time to test and I have a corrupt drive i'm using right now so i havent been able to get very far.. Do you think my theory has any chance of working??


It might work - but that's getting beyond my Unix expertise, which isn't that great. If your system supports xfs, intuitively one would think that the gui would see the whole file system once you mount it properly. If all you want to do is browse the file system, you may have an opportunity. If you want to actually do the file transfer, I would not be optimistic.

Actually, I should be clear that I don't use GPartEd to mount the drive - the mount command we use is basic Unix, as long as you have a flavor that supports xfs. I just happened to be using GPartEd because that's the Unix that has been known to work for copying an entire drive.

#16 OFFLINE   UhClem

UhClem

    Cool Member

  • Registered
  • 22 posts
Joined: Oct 01, 2009

Posted 21 August 2012 - 12:51 PM

[Yes, I'm late to the party ... (I visit dbstalk infrequently and missed this -- just now happened upon the thread via a somewhat "misguided" Google hit)]

...
Another caveat that I have to make clear - if a recording has been transferred from one drive to another, the time stamps on all the transferred files will be the times they were written to the new drive, not the time stamp on the original.

You can use the touch command to "restore" the (original's) timestamps.

#17 OFFLINE   mocarob

mocarob

    Legend

  • Registered
  • 186 posts
Joined: Jul 26, 2007

Posted 21 August 2012 - 01:52 PM

[Yes, I'm late to the party ... (I visit dbstalk infrequently and missed this -- just now happened upon the thread via a somewhat "misguided" Google hit)]

You can use the touch command to "restore" the (original's) timestamps.


Whats the 'touch' command?

#18 OFFLINE   P Smith

P Smith

    Mr. FixAnything

  • Registered
  • 19,324 posts
  • LocationBay Area
Joined: Jul 25, 2002

Posted 21 August 2012 - 01:59 PM

Major question: how is it important to restore timestamps ?

#19 OFFLINE   unixguru

unixguru

    Godfather

  • Registered
  • 489 posts
Joined: Jul 09, 2007

Posted 22 August 2012 - 01:54 PM

Add the '-p' (preserve) option to all the 'cp' commands. This will carry the timestamps over to the new files.

There is a -p option to xfs_rtcp but it is something entirely different, don't use it for this purpose.

'touch' is a unix command that takes a timestamp specification and applies it to file(s). See man page for format. This can be used to change the timestamps on the realtime files if it's needed.

#20 OFFLINE   cigar95

cigar95

    Legend

  • Topic Starter
  • Registered
  • 135 posts
Joined: May 23, 2008

Posted 24 August 2012 - 01:07 AM

Major question: how is it important to restore timestamps ?


Wow, this thread is back?

Anyway, restoring timestamps has NO importance in terms of the watchability of the recordings. It only comes into play, as far as I can tell, if you are trying to trim off unneeded pieces of the recording by deleting segment files at the beginning or end of the recording (NOT in the middle!), and you need to know when different pieces of the recording were actually made so you know which segment files to delete.

As for using the "touch" command - these recordings may include hundreds of segment files. To use it for preserving time stamps, it would essentially require some fairly good facility with Unix scripting. I'm sure our local Unix gurus would find this a simple process. But I'm not one of those.

A note to our board mods - if this topic broaches the limits of acceptable discussions on the boards, please contact me before deleting it so I can be sure to preserve a copy for my own use. Thanks. I'm not sure I have documentation this complete anywhere else. (I don't plan to do this any more if I can avoid it - I've found that there is a non-trivial chance of corrupting a drive just by something going wrong in the process of mounting and dismounting it in Unix. I would not do it simply as a convenient way of moving a recording between my external drives, even ones that both work on the same DVR.)




spam firewall