the threat of E* guide compression

[jhall]

With the new year upon us and Charlie's threat to compress the guide again, I have decided to ask some questions of people owning legacy receivers, and those that do not have a hard drive.

1: When exactly do you see the performance issues?
A: when pressing the guide button
B: when pressing the more details button on some program to get a more detailed description
C; when using the feature that displays the guide along with the current program
D: all of the above
E: none of the above

2: When the guide was briefly compressed, for a week or two, did you notice a difference?
A: all the time
B: most of the time
C: some of the time
D: none of the time
E: they compressed the guide?

3: perform whatever action takes the time on programs on the following channels: rate the responsiveness and whether you think things are normal, slower than normal, or faster than normal. Be sure to perform multiple tests and see if you get reoccurring results.
A: for channels 124, 129, 184, 191, 204, 206, 272, 300, 432, or 455 (you do not need to do all of them)
b: for channels 118, 120, 137, 173, 200, 303, 320, 635, 636, 955, 964, 965, 977, 978, 979, or 982
C: for channels 100, 217, 223, 506, 510, 511, 516, 521, 523, or 9611
D: for a spot beam you have LIL

I have more questions, but this should get things rolling

[scooper]

1. A - what I typically see (4700 and 2700 here, both at latest 4900 / 2800 FW respectively) is the first hour / 1.5 hours (2 entries for each channel in the EPG) it gives the proper Guide data, for all after that it say "No Info".

2. E - I did notice for a time that I had more entries further out.

3. I'll check later..

[normang]

If compressing the guide improves perforamance and reliability of the guide for non-PVR receivers, why is this some sort of threat?

Are we sure that the guide has even been compressed yet? And even if it has and the initial code and compression has not worked as hoped, does this mean they shouldn't do it at all. Like anything, It would be nice if everything worked perfectly the first time, but rarely does this happen in my experience.. usually takes a few revs of code to workout issues, no matter what is is..

[jhall]

Compressing the guide would mean it is not visible to an FTA receiver. I rely on the FTA guide to be able to know what is on and when. These questions are designed to offer alternatives to E* Engineering (if they will listen)
They've probably thought of my ideas before, but I want to at least try. The DVB spec says a DVB-compliant device should be able to withstand tables and sections at 100ms intervals. I haven't calculated the time between sections, perhaps it is as simple as increasing the data rate.

[Jacob S]

So what is the advantages of compressing the guide anyways? It does not take up that much space on the satellite does it? Is this their way of preventing people from getting free things on FTA receivers? This is a crock of bull if it is.

[RJS1111111]

I'd be happy if the "please wait or cancel" option
never appeared, and guide navigation would never
be frozen in this manner. You could then navigate
smoothly to the channel and time range of interest,
then wait for "Info not available" to get filled in
with actual guide information, or continue to
navigate anywhere else you like.

The software should also be smart enough to
detect your current general direction of navigation
(to the right in time, channel up or down), anticipate
what will need to get filled in next, and ignore
all of the other, unwanted guide information.

There should also be some user interface feedback
provided about how long the last complete guide
download took, and how long your current wait
is likely to take.

The guide should also take much longer to
time out back to live TV; say at least two
complete guide downloads.

This, I think, could make guide compression
completely unnecessary, and make the use
of legacy receivers a much more pleasant
experience.

Yes, if necessary, I would be willing to pitch in
a few bucks for a memory upgrade.

[scooper]

So would I if it was easily accomplished by consumers. I'm not putting a soldering iron in my IRD's...

[jhall]

They want to compress the guide because they say it will take half the time to roll the guide if it is compressed. They could recover some bandwidth if they didn't show guide entries for nonexistant channels. Perhaps one of you could describe the difficulties you are having and how to trigger them. I would not be as noisy about it if E* would tell us how to uncompress the guide before compressing it.

[Jacob S]

so if they just download the guide data for the channels one is receiving and compress it on top of that it would bring about a quicker viewing of the guide? are they talking about the channels not subscribed to or just the locals in your area?

[jhall]

They are going to compress the datastream, in effect reducing the amount of content needing to be transmitted from the bird. I have heard it takes in upwards of 30 seconds to transmit the whole guide.
In short, on all the transponders, on a "guide channel" E* transmits all names and descriptions for programs on all channels in the bouquet. It has the added annoyance that items for programming on 129 and 148 are viewable for 61.5 users, as well as items for 61.5 users shows up on 148.
On a "special" transponder, EPG information for two days is transmitted. Can you make a legacy receiver go more than 2 days in the future?
It appears the bottleneck is in the amount of information that must be transmitted at the current rate. If an E* receiver is multithreaded, the data rate might be able to be increased and the receiver remember where in the datafile it was when it started listening and stop next go around, so it sees a single iteration fo the file. The other part of the receiver could be parsing the data that has come in

[Nick]

Or, get a PVR. Instant everything, 7-9 days out. Cool. :)

/Legacy PVR owner (2).
"Wouldn't go home without them."

[bunkers]

I am hoping the move to the compressed EPG will allow nightly refresh(es) of the the EPG information (if they don't already). I see this a a big part of the timer-slot based timers in the DISH PVR(s).

[jhall]

the guide that the PVRs use appears already to be compressed.

[Jacob S]

In the past the guide was not this bad that I recall. It would come up right off the bat and you could go a day in advance before ''no info'' would pop up.

[Scott Greczkowski]

Here is my thoughts on the compression thing (and I could be wrong this is just my understanding)

I think one part of the compression is currently they have lots of channels because of Must carry that are showing the same thing.

For example on NBC Monday Nights Fear Factor is showing on 53 NBC's and the description for that show are being sent separately to all 53 NBC stations. Thats a lot of information being sent out.

With compression they will just send out the description to Fear Factor once with some encoded channel information, (instead of sending it out 53 times) yet in your guide it will show up just as it has before.

Thats an incredible amount of savings in the data being sent and received.

Plus at the moment your receiver downloads the guide data for every channel Dish Broadcasts, with compression and some other things they are working on your receiver will only download the channels you receive and disregard the rest.

Ultimately with compression it means for better guide operation for everyone.

Of course if it was not for Locals In Locals I don't think they would be having the guide problems they are having now.

[DmitriA]

I hadn't thought of that Scott. I assumed they would use standard data compression techniques (ie. zlib or something like that), but your way is probably going to get much better results

[jhall]

In order to do your first recommendation, i.e. reducing the amount of times the same program shows up they would need to use their own proprietary guide format and either ignore or move away from the DVB standard for EPG. The compression they used was data compression for the title and description (0x91 and 0x92)
In short, every service id needs to have guide information for it, so for example, WJLA is 8070 and 700. The schedule must be transmitted for both 8070 and 700. The DVB standard has previsions for time shifting, which E* does not appear to be using (hbo-e and hbo-w for example)
I need to look at the standard more closely, but I THINK it is possible to reference an event by event ID. If they could do that, they could get some savings.
yes, this comes from vdr source code.
case DESCR_TIME_SHIFTED_EVENT:
{
struct tm *StartTime;

if (!VdrProgramInfo)
{
CreateVdrProgramInfo(VdrProgramInfo,
Event->EventID, Event->TransportStreamID,
Event->ServiceID, Event->StartTime,
Event->Duration, Event->Status);

VdrProgramInfo->ReferenceServiceID =
((struct TimeShiftedEventDescriptor
*)Descriptor)->ReferenceServiceID;
VdrProgramInfo->ReferenceEventID =
((struct TimeShiftedEventDescriptor
*)Descriptor)->ReferenceEventID;
}
}
break;
As you can see, they could very easily use this mechanism to transmit the events once then reference them on all these duplicate channels.
So my recommendation to E* engineering, even though they have not asked me, is that they begin deploying this approach and avoid using compression, merely use the time shifting mechanism. You've got 65,535 service ids, why not start at 64512 (localscope) and begin numbering service ids for networks, NBC, CBS, ABC, etc. If you don't want to use a precious service ID for this, simply allow the first channel at a given time or in the future to reference the event directly, all the others use these time shifted event descriptors. I am STRONGLY against using data compression as a mechanism to make the guide appear faster.
I would also recommend extending the amount of events available on each normal transponder, i.e. using more than 4E and 4F. Because descriptions will not be transmitted for an event more than once, it should not be as big a problem to transmit 2 or even more days in advance on normal transponders. The more days in the future, the better chance an event could be sent in the stream only once. For example if Enterprise airs on Thursdays at 01:00 GMT, the "master" UPN schedule could run it at 07:00 GMT on Friday, giving the locals a full 24 hours to air Enterprise. This works well for syndicated programs that air at some point in a 24 hours period, like Voyager, that airs on many UPN, fox, or independent stations through the Paramount network. As long as the master event is in the future and is being transmitted on all the transponders, the receivers should have no difficulties with it. The only time this might be a problem would be if each station decided to run fully independent programming that is ...

[jhall]

... fully unique to that station and will not repeat on another station. It is sad that most of our television is a copy of somebody else's work. Timeshifting is the standards way of solving this problem, I recommend it be used instead of compressing the data.

[jhall]

in fact, as a trial run, I would ask E* to begin testing with one or two of the 7xx and 8xx channels for the time shifted event. In effect, this would greatly reduce the amount of guide information that need be transmitted. Pick something on a transponder where all the events for that transponder fit into a single section (to prevent the race condition of missing the reference service id's events on the first pass)
All I ask is that my words are heard by E* eng. an anonymous ``your words have been heard'' would do as confirmation.

[Darkman]

hehe,
some lamer can just type that :) - to make you happy - lol

[jhall]

yeah, but I'm hoping for honesty here. I'm also hoping that the reply that says
your message has been sent
would also come with a reply packet that says
your words were accepted and plans exist to implement
or
your words were rejected because ${VALID_REASON}
I'm trying to promote happiness between the clueful customers and Engineering.

[zimm0who0net]

jhall,
What's your beef with compression? It sounds to me like the best approach would be to use your indirection suggestion and then compress the data as well. Depending on the compression, you would probably kill any FTA receivers who actually use those to retrieve the extended guide data, but there can't be more than just an extremely small handfull of those out there.

[jhall]

Because the guide is ITC, I can have the computer read it to me and thus enjoy the guide as sightlings do through the E* receivers. Now that I have it, I couldn't live without it. I saw only about a 1/3 improvement if that when the guide was compressed.

[jhall]

FYI EPG compression appears to have started around 14:00 EST, 19:00 GMT. Does anybody notice any speed improvements? Does anybody notice a dramatic change that justifies basically crypting the guide?

[scooper]

WOn't get home until after 6 tonight - will check and post.

[jhall]

They appear to have switched it off, perhaps it was a quick field test, perhaps they found some problems, perhaps they wanted to see just how much better it really was live. Why they would test in the middle of the afternoon as a "hot cut" is beyond me.
I'd expect to see compression in limited testing from time to time, then one day activation.

[Jacob S]

Would this have caused the signal outages?

[jhall]

I doubt this would cause a signal loss, but if it caused your receiver to coredump, that could cause an unhappy experience for viewers. If you specifically had signal loss between 14:00 and 18:00 Eastern time, 19:00 to 23:00 GMT, then you might be able to make a case for it. I hope it did. :)
but I don't wish any LOS upon E* customers.

[jhall]

well, it looks like guide compressing began full time last evening around 19:00 EDT. Anybody noticing HUGE DRAMATIC speed improvements on guide surfing?

[Darkman]

I am not even sure, it is hard to say (on my 4000 anyhow),
my 3900 is busy at the moment (recording something)....
In any case i didn't see nothing DRAMATIC on my 4000 anyways (if anything at all)...
And didn't they say at Tech Chat - about compression - that it will begin Next week (like Wednsday or something) and not this week?

[jhall]

they said wednesday, and well wednesday was yesterday. They are definitely compressing.

[Darkman]

hmm - i see - BobaBird's recap says on Wednsday 02/12 - but somehow i thought i saw and heard that they said on Tech Chat - next Wednsday, not this one...

Anyone else - what did you hear about it on Tech Chat? - do you recall?

[jhall]

yeah I saw the tech chat replay (I guess it was replay) last night and by the time the replay was over I saw that my guide had compressed. They still claim a 2x1 compression and thus faster guide times. I'm anxiously awaiting reports from folks on the VAST speed improvements. should be halving, right?

[Darkman]

jhall, hehe -then you heard for sure them say Wednsday 02/12 or what? ( or it could have been next wednsday anyhow? :) )
Too bad i didn't tape it - or could have re-checked now for myself :)

[BobaBird]

I thought they meant Wednesday of this week, so I added 2/12 to allow for people reading it next week.

But then again, I'm in an all-DVR household so I don't really care anymore how big the guide is! :ducks and runs:

[Darkman]

i see...
as to the "Wednsday" matter - i am still pretty sure he said Next Wednsday :)
Hehe - didn't any1 hear or remember what he said?
Didn't he say something like: "This week 508s or 501s, and next Wednsday we'll start sending the compressed guide, 2:1 ratio, and we even have more tricks up our sleeves :)"

Or something like that? NO?

[Jacob S]

They said they had more tricks up their sleeve and that they may even have one guide for the locals and another one for the other channels. They said that the must carry law is what caused the guide to load a lot slower than it used to because of all the data needed for all those channels.

Since when did the 3900 record?

[Darkman]

3900 records? - huh - what are you referring to? :)

{EDIT} - oh i see - you were refering to my 3900 - hehe - it wasn't Busy recording, it was just Busy (while my VCR was recording) ..therefore was busy to check it's EPG at that time :)

as to other stuff - tricks, sleeve, etc - Yes that is what they said :)
but getting back to the "Wednsday issue" - Do you or any1 else recall them say Next Wednsday? or Did you hear 2/12?

[jhall]

well, the way I see it, it doesn't matter whether he said Wednesday or next wednesday.
The context I recall was PVR508 is going out this week, possibly even said early this week, and then guide compression, which will provide a 2x1 compression begins on Wednesday. My understanding as I heard it was it would be 2/12.
But it doesn't really matter, because it did promptly begin compressing at end of day on 2/12. For the E* people who are reading this, can we compromise and:

A: pick a transponder that has unpopular channels on it and transmit the uncompressed guide for 2 or more days on it

B: for the 2day guide (on tp19) transmit compressed, but transmit uncompressed for now&next (4E/4F) on the other transponders

C: release a not-supported whitepaper explaining how compression works

now with regards to splitting the guide so CONUS gets one guide and spot gets another, how about:

transmit 2 or more days for channels on a spot on that spot in the normal EIT sections, for example TP266 in network 4102 could show all the PHL channels, plus any FTA channels that happen to be present (as a courtecy)

caveats:

This MAY require a software update, but you could use the linkage descriptor possibly to point receivers to the right thing.

It means you cannot just generate a flatfile for the whole network, rather you would need to customize files for each SPOT TP

As far as must carry requiring the guide to be so big, is there a must carry clause that says you cannot reference programming from some other network or internal service to ease in delivery? I.E. do must carry rules prohibit use of timeshifting directives?
If it is determined that guide compression is not providing a signifficant win, can we go back to uncompressed?

[Darkman]

hehe - jhall, you sure like to get technical and to the point :)
as to Compression - how do i test it - i tried to check couple things last night, but didn't notice anything Dramatic or Drastic (if anything at all) (re: improvements) at my 4000 and 3900 anyhow...

Maybe if it was sent - the compression - it was sent not to Everyone?

[jhall]

no, it is sent to everyone. If it was taking o the order of seconds to pull up your guide, you were supposed to see improvements. If your dish machine has a hard drive in it you will not notice a difference. This is supposed to fix legacy users. I am looking at guide entries for SCIFI and I have schedule info but not program and description.

[dlsnyder]

Last night I was checking the guide on my 3900 for a program airing today (about 12 hours from the time I was checking). The guide downloaded much faster than usual, at least twice as fast. Looks like guide compression works for me :)

Now I will have to try the same thing on my 2700 just to see what happens.

[Darkman]

nice to hear dlsnyder...
but 3900 is faster then lets say 3000, 4000 and such (anyhow)

I would like to hear your opinions on 3000s, 4000s, 2700, 1000, etc...

[JohnH]

There is a difference on the 301 in the way it downloads the guide. The blocks are different in their spacing.

[Lee L]

My 6000 still takes at least one minute to get the next 2 hours of data beyond what is on right now.

[lee635]

The guide is working much faster on my model 3000. We're actually using the info button again!

Great job E* techs.

[sampatterson]

My 6000 would usually not show more than 1 to 2 hours in the guide(no information), now it has around 6 hours, so they must be compressing.

[Darkman]

well, nice to hear,
But i haven't really noticed anything drastic on my 4000 (or even 3900) - maybe i should look at it / play with it more closer and see if THEN i notice anything exiting :)

[scaredpoet]

Downloading the guide on the 301s I have (yeah, I know, I need to upgrade) has changed noticeably. It's gone from unbearably slow (up to a minute, sometimes a little longer), to moderately annoying (30-45 seconds).

Does anyone know how large the program guide is data-wise for, say, two to three hours worth of programming? If you go by the website and pull programming data for three hours on all channels, the size of the resulting HTML file comes out to 568k.  Is the version coming off the bird that much bigger?  It just seems odd to me that a dbs receiver can pull streaming MPEG2 video and audio without a problem, but a program guide takes forever to update.

[JohnH]

There is approximately 1226 channels in the system. A few do not have guide info. Consider how much is required to name the program, fill the info screen, provide ratings, start times and length. Multiply to see the minimum amout of space. there are other items in the stream as well.

They do not assign much band width to it, since the info has to be processed after receiving.

[Darkman]

Well, finally today, i had a chance to play around somewhat with my 4000's Guide - going forward on it, etc - and it DID seem faster! :-)

[kstuart]

Compressing the guide would mean it is not visible to an FTA receiver. I rely on the FTA guide to be able to know what is on and when. I'm confused - where in the description of service does it say that FTA receiver support is included?

[Mike123abc]

The 6000 seems to be holding about 4 hours now. And, when you go past the 4 hour mark it says retrieving data from satellite... and sure enough in about 45 seconds you can get the next 4 hours!!! Now it is not the best browsing past 4 hours because it keeps having to go back to the satellite, but at least it seems to work now. It seems to get 3 hours at a time past the current block.

[jhall]

kstuart writes:

<Quote>
I'm confused - where in the description of service does it say that
FTA receiver support is included?
</Quote>
I admit I may be considered on shaky ground when I say this, because the ITSE standard does not specifically spell out in the main standards document how an EPG should be written, it doess provide methods for dealing with the overstuffing problem that are standards-based. E* is choosing not to follow the standards by compressing their guide and not providing an explanation how to decompress it. Until a week and a half ago, they were following the standards as far as how to transmit the guide. True they were not implementing some of the standards, like genre or rating descriptors, but at least the guide was usable.
If you believe a guide is a schedule of events, then they are still complying with standards. If you believe those events should have names or descriptions, then Echostar Communications is no longer complying with the DVB spec IMHO.
So it's an old-school philosophy that is being swallowed by a proprietary closed-source one.
Personally, I think if a user can obtain a receiver with a CAM that can be authorized, at least minimal efforts should be made to authorize the CAM. A person might have a plausible reason for it. I have no idea how the CA system actually works, so it might not be technically possible to authorize a foreign receiver.

[Lee L]

I have also noticed the 45 second time on my 6000. I guess when I tried it before, I had left the 6000 on an OTA station for a day or so and it had nothing in the guide at all. Once I left it overnight tuned to HBO HD, I had much better guide performance. Too bad they can't figure out some way to let it keep getting the stream while tuned to OTA.

[Scott Greczkowski]

I think when you are in OTA mode it stops looking at the satellite signal as the tuner somehow is switched to the OTA module.

Thats why after watching OTA for a long while you press guide and get the "Aquiring Satellite Signal" message.

[Mike123abc]

The next upgrade to the 6000 I wish they would get around to is changing the guide from just saying "Local Programming" to the show name since it actually knows the show name (you can press info and it gives you a nice description). During prime time it will show the show name, but the info works all day.

[JayeDVXIII]

HEY jHALL

does your FTA receiver even have a guide?

[jhall]

well, depends on what you mean by guide. It will load the times but the title and description says (NULL) because there are no longer decompressed titles and descriptions visible. For example, I can tell that a program on a channel has started at some time and the length, but not what it is.