Editing LDC File Format Specification

=====LDC File Format Specification=====
The *.ldc file format is a decklist format used exclusively by the [[Lord of the Rings Online]] software produced by Decipher and Worlds Apart. It is a proprietary format, recently analyzed and picked apart to facilitate conversion of *.ldc decks into more modern and useable formats.

A collection of decks that entered various tournaments can be found [[http://lotrtcgwiki.com/forums/index.php?topic=8730.new|in this TLHH post here]] if you'd like to follow along. If you intend to write a parser, be sure to get the copy of LotR-O in that same post; the various installers floating around do not include any updates and so only contain sets 1-13.  


====Format Quirks====

There are a few aspects of the file format which are a bit strange, but at least are consistent.  The two biggest to keep in mind are the string format and length format.

===D-String===
All string information is kept in a bizarre blend of a Pascal string and a C string.  For those unfamiliar, a Pascal string is a string which reserves the first byte for length information; it's fast and relatively error-free but each standard Pascal string is limited to 255 characters long, which is the maximum size you can record in 1 byte of information.  C-strings, on the other hand, record no length information and simply keep going on and on until the null terminator (binary ''0b00000000'', hexadecimal ''0x00'') is found.  This allows strings of an arbitrary length, but is more intensive to analyze, since you never know just how long your string can be.

Strings within the *.ldc file format have //both// a size //and// a null terminator.  It's possible that the null terminator was intended to be a padded byte between entries, but considering it is always in place and does not align other bytes at all, I find it unlikely.  At any rate, I have taken to calling this style a D-string.  The size is a dynamic 1- or 2- byte length explained in the Length Format section below.

An example of a D-string is below, taken from one of the tournament decklists:

   0a  42 72 69 61 6E 5F 46 72 65 64 00
  [10] B  R  I  A  N  _  F  R  E  D  \0
  
===Length Format===
Bytes representing length of information to follow (as in the D-string above or in card arrays) follow a dynamic 1- or 2-byte format (higher combinations are possible, but I wasn't willing to manually create a deck of 4,096 cards to test this).  

If a length byte is less than 16 (so from ''0x00'' to ''0x0f''), it will remain one byte, but as soon as it breaks 16 or more it branches into 2 bytes.  The reasoning behind this is a bit obscure; since a single byte could keep 255 values, it's uncertain why you would need capacity for more than that in the deck specification.  For context, you would need a deck size of 511 cards (255 Free Peoples, 255 Shadow) or a deck name spanning 511 characters to ever break the one-byte limit, but it seems someone at Decipher or Worlds Apart was worried that was too small, but didn't want to commit a 2-byte length everywhere.

As a further weirdness, the first byte cycles between the same values; it is always between ''0x20'' and ''0x2f'' (32 - 47), and the second byte modifies the first. The total actual length can be calculated as follows (in decimal):

  byte1 + (16 * (byte2 - 2))

An example of a 2-byte length, also taken from the tournament files:

  22 01 4A 6F 73 68 5F 43 6F 72 6E 77 65 6C 6C 2D 4D 6F 74 74 00
  b1 b2 J  o  s  h  _  C  o  r  n  w  e  l  l  -  M  o  t  t  \0
  
''0x22'' in decimal is 34, and ''0x01'' is 1, so if you plug it into the formula above, you get 34 + (16 * (1 - 2)), which reduces to 18, the number of characters in the string "Josh_Cornwell-Mott".

It's important to remember that this format is used for all length bytes within the *.ldc format, even if it seems superfluous; if it is not taken into account, you are likely to run into problems where lengths of 0-16 and 32-47 parse correctly but all others do not.

It's also important to understand how the length format works; it's used not just for reading arrays but also in determining the ID of a card.  This piece of information, when it finally clicked, was the key to unlocking the whole file format.

====Deck Format====
The format of the deck is very straightforward; no obfuscation or encryption of any kind (this is the job of the card ID system, which is appropriately obtuse).  The major parts of the file are as follows:

  * File Signature - 3 bytes
  * Deck Name - 2+ bytes
  * Format Style - 2+ bytes
  * Unknown Bytes - 5 bytes
  * Ring-bearer - 2+ bytes
  * The One Ring - 2+ bytes
  * Free Peoples deck - 1+ bytes
  * Shadow deck - 1+ bytes
  * [[Adventure deck]] - 1+ bytes
  * Starting Fellowship - 1+ bytes
  * Footer - 2 bytes

A brief discussion of each section is below.  All bytes are shown in hexadecimal format unless otherwise specified.

===File Signature===
Every deck tested so far has the same preamble or file signature, so it's unlikely that versioning information is kept here (or perhaps just that an additional version was never created and supported).  These bytes are the following:
  2E 02 03
  
LotR-O will claim the file is corrupted if any of these three are changed, so it is at least enforced.

===Deck Name===
The deck name is included as a D-string, which presumably has to be at least one character long (but I have not tested this). The deck name is taken from whatever the file itself is called when you save it within LotR-O.  This allows renaming of the file while the deck name itself is preserved within the format.
  
===Format Style===
The next piece of information I have labeled "format style" for lack of a better name.  It is a D-string, which in most of the tournament decks is a four-digit code of some sort.  All new decks saved have the word "default" here, but it's unclear if this is supposed to represent the author, the tournament format, or some other information.  At any rate, it's not saved between incarnations; if one of the tournament decks is loaded into LotR-O and then resaved, the four-digit code will be replaced with "default".  

Occasionally this information is missing (length of zero) in the tournament files.  Don't forget to grab the null terminator byte that always follows, even in this case.  If your D-String loader is operating according to the specs described in that section, this won't be a problem.

===Unknown Bytes===
Following the format style are five null bytes (so six in a row if you count the null termination of the Format Style d-string).  Their intention is unclear; if it's just padding then it makes no sense why 5 bytes would be chosen over any other number.  It's possible (and perhaps more likely) that it represents five unknown pieces of information that all default to ''0x00'' when undefined.  At any rate, if they are additional information, it's all meta information, as they have no bearing on the deck itself.

Some tournament decks do indeed have D-strings embedded in these bytes, but as they are always copies of the Format Style (or replacing the Format Style when it's blank), it's still unclear what information these bytes are meant to represent.

===Ring-bearer===
The Ring-bearer of the deck stands alone outside of any card arrays, which sort of makes sense as you are limited to only one ring-bearer per deck (this, along with the Ring, are the only restricted parts of deck building; you can easily make decks with uneven Shadow/Free Peoples sides, or 40 sites, for instance).  

The ring-bearer is recorded in the same format as all other cards within arrays, so see the Card Format section below for a description.  If no ring-bearer is present, there will be two null bytes (''0x0000'').

===The One Ring===
The One Ring is recorded in the same manner as the Ring-bearer, and like the ring-bearer will instead have two null bytes if not present.

===Free Peoples deck===
The Free Peoples deck marks the first of four card arrays, which are all recorded in the same straightforward style.  

Each array is prefaced by the length, which is the dynamic 1- or 2-byte length described [[ldc_file_format_specification#length_format|above]].  The length describes the number of cards it contains, and not the number of bytes (as this varies from card to card).  There is no null terminator at the end.  If there are no cards in the array, a single null byte (doubling as the length of the array) takes its place.
  
===Shadow deck===
The Shadow card array is identical to the Free Peoples array, and follows immediately with no padding.

===Adventure deck===
The Adventure deck is identical to the previous two card arrays.  It should be noted that there is no sanity checking, so mixing sites from different blocks is possible, not to mention having many more sites than the 9 allowed by the game.

===Starting Fellowship===
Oddly enough, the starting fellowship is treated as an entirely separate deck, so if it is defined it will duplicate, rather than replace, those companions from the Free Peoples deck.

===Footer===
At the very end of the file are two null bytes.  These may just be filler, they may be additional unknown meta information, but regardless LotR-O checks for them and will reject the deck as corrupted if they are missing.

====Card Format====
Here is where things get tricky.  All of the previous information was very straightforward and identified in less than an hour of tinkering, but the format that cards are saved in is quite convoluted.  As with most systems of this complexity, it's possible there's just one or two pieces to the puzzle missing that would force it to make sense, but with the current understanding it is at least possible to parse.

Cards are saved in a format that varies from 1 to 4 bytes in length, followed by a null byte (so it could be considered 2-5 bytes long, but that's semantics).  

The format of cards from different sets varies, which makes one think that whoever came up with the format did not plan ahead very well, and as new sets were released new formats had to be devised that were backwards-compatible.  It's a mess, quite frankly, and some decisions don't make any sense whatsoever (such as the fact that foils are contained within their own sets, rather than as a flag on a regular card).  

LotR-O contains information on all 19 sets plus most 0-set Promotional cards, as well as plenty of W-series cards that were never hosted in the database on Decipher's site.  If you use the installer to install LotR-O, you will be limited to information for cards up to set 14 (//Expanded Middle-earth//), though the effective limit is set 13 (//Bloodlines//), as the user is not given access to set 14 cards in that version.  Be sure to get the files from the post at the top of this article for maximum information. 

Descriptions of each style of card are given below.

===Promotional===
Promotional cards, for whatever reason, appear to be the proto-format, which makes no sense as you can't play a full game with them.  This format is almost exactly the same as the length format, both in form and short-sightedness.

Promo cards from 1 to 15 are simply straight-up numbered in hexadecimal: ''0x01'' to ''0x0F''.  Starting with number 16, the format changes to a 2-byte setup. For all 2-byte promotional cards, the first byte ranges from ''0x20'' (32) to ''0x2F'' (47), and the second is a modifier following the same formula as the length format:

  byte1 + (16 * (byte2 - 2))
  
The number produced by the formula is the card number.  For identification purposes, if the second byte is less than ''0x24'' (which is where //Fellowship// cards start), then it's a 2-byte Promotional card.  Cards which are 2-byte are not sequential; most of the first 60 are 2-byte, and then many in the 100 range are also 2-byte.

Following card 61 (The Witch-king, Black Captain (P)), the card coverage shifts to using the 3-byte pattern of Fellowship/Tengwar, below, when the format is not 2-byte.  Such cards have a Identifier Byte of ''0x49''.

===Fellowship/Tengwar===
Cards in set 1 (//Fellowship of the Ring//) follow a 3-byte pattern that echoes that of the length format and provides a foundation that all other cards are based on.  The three bytes are as follows:

  * A //Repeating Byte// that loops through the values between ''0x40'' and ''0x4f''
  * A //Sequence Byte// that is equivalent to the modifier byte of the length format
  * A //Identifier Byte// that is the same for every card within the //Fellowship// set (but not the block)

For instance, The One Ring, Isildur's Bane is recorded as:

  41 24 F4  
  
It is the first card in the set, so its Repeating Byte is ''0x41'' (sets universally start their numbering off at one, leaving a potential "card zero" empty in each set).  

Its Sequence Byte is ''0x24'', which it will share with the next 15 or so cards, and is a multiplier for the Repeating Byte (the formula however changes with each set, as sets are not sequential with one another).  

Its Identifier Byte is ''0xF4'', and it shares this byte with every card in set 1, and is the primary way of differentiating between sets.

(Not for lack of trying.  Starting with set 2 (//Mines of Moria//), there is a fourth byte added that records the actual set number, but let's not get ahead of ourselves).

Tengwar cards form their own set, but only include cards from set 10 (//[[Mount Doom]]//) and earlier.  Cards within this set have an Identifier Byte of ''0x02'', but are otherwise identical to the organization of the Fellowship cards.

Cards with a rarity of W and D also utilize this 3-byte format; their information is given in the table in the Set Organization section.

===Standard===
All other cards including sets 2-19 and their foils follow a 4-byte format.

To a programmer, the organization of the 4-byte format is particularly hysterical, and it tells a compelling story worthy of any made-for-TV movie.  Take a front row seat as you behold the struggles of the Worlds Apart programmers as they realize that their previous formats were garbage, they add new information to compensate, and then subsequently fall beneath the weight of their own incompetence.

(This was a real blonde, brunette, redhead moment for me as I watched it unfold, let me tell you.)

The "standard" 4-byte format is an evolution of the previous //Fellowship// format, basically copying the 3-byte structure and adding a 4th byte that was apparently supposed to be a set number.  For this example, I'll use Beneath the Mountains, the first card in //Mines of Moria//:

  61 48 E8 01
  
''0x61'' is the Repeating Byte, once again starting with 1 instead of 0, as it's the first card in the set.  The primary difference between the standard format and the Fellowship format is the fact that the Repeating Byte ranges between ''0x60'' and ''0x6F'' instead of ''0x20'' and ''0x2F''.  The reason for this is unknown, but appears to be arbitrary.

''0x48'' is the Sequence Byte and appears to be greater than the highest card in //Fellowship//, but there's quite a large jump, leaving a lot of unused sequences.  

''0xE8'' is the Identifying Byte, which it shares with all other //Mines of Moria// cards.

''0x01'' is what could be termed the !Set Byte, and it appears to be an attempt to be an //actual// representation of the !set number.  Of course, it's off by one--//Fellowship// is set 1, but //Mines// gets the moniker of set 1, giving //Realms of the Elf-lords// !set 2 when it should be 3, etc, all the way down the line.  The victory is eroded further, as foil sets also have the !Set Byte, so your //Siege of Gondor// foil has the same !set number as your //Siege of Gondor// nonfoil, so you can't rely on the !Set Byte to differentiate cards anyway.

But wait--there's more!  Foil sets were apparently not in the original client release, and when they were added they were all saved in 4-byte format.  This was apparently an opportunity to fix the !set mismatch, and so //Fellowship// foils have the proper !set 1 attached, while //Mines// foils have the !Set Byte !set to 2, and so on.  However, the fact that the real cards were "wrong" and the foil cards were "right" was apparently too much to keep track off, so around the release of the //Return of the King// foils, the sets were matched up, so both regular cards and foil cards from the same !set had the same !Set Byte.  

//This means that both foil //Return of the King// and foil //Ents of Fangorn// have the same !Set Byte.//

To recap: 
  - Regular cards after //Fellowship// all have the wrong off by one !Set Byte
  - Foil sets of //Fellowship// through //Ents// have the correct !Set Byte
  - //King// foil sets onward have the same off by one !Set Byte as regular cards
  - And to top it all off both foil //King// and foil //Ents// have the //same// !Set Byte, one wrong, one right.  

So, to make a long story short, the !Set Byte is worthless.  The amount of work one's code would have to do to keep track of all this is not worth it, especially when you have to use the Identifier Byte //anyway// because of the overlap between //King// and //Ents// and the overlap between foil and nonfoil.  

====!Set Organization====
So now we've seen how decks are recorded, and how cards are recorded within decks, but how best to use this information?  First we need to gather some bird's-eye data.  I've taken the liberty of grabbing the major information that defines each set; this information was obtained by making a deck with one card from each set, with that card being one of the first 15.  This gives us the set's Identifier Byte as well as the Sequence Byte for the first group of 16.  All of this data is below:

<WRAP indextab>
^  !Set Name  ^  !Set #  ^  First Sequence Byte  ^  Identifier Byte  ^  First Foil Sequence Byte  ^  Foil Identifier Byte  ^
|  Promotional  |  0  |  -  |  -  |  0x42  |  0x49  |
|  Tengwar  |  ~  |  0x71  |  0x02  |  -  |  -  |
|  W-Series 1  |  W  |  0x71  |  0x18  |  -  |  -  |
|  W-Series 2  |  W  |  0xa8  |  0x61  |  -  |  -  |
|  Fellowship of the Ring  |  1  |  0x24  |  0xF4  |  0x62  |  0x3D  |
|  Mines of Moria  |  2  |  0x48  |  0xE8  |  0x86  |  0x31  |
|  Realms of the Elf-lords  |  3  |  0x6C  |  0xDC  |  0xAA  |  0x25  |
|  The Two Towers  |  4  |  0x90  |  0xD0  |  0xCE  |  0x19  |
|  Battle of Helm's Deep  |  5  |  0xB4  |  0xC4  |  0xF2  |  0x0D  |
|  Ents of Fangorn  |  6  |  0xD8  |  0xB8  |  0x16  |  0x02  |
|  Return of the King  |  7  |  0xFC  |  0xAC, 0xAD  |  0x3A  |  0xF6  |
|  Siege of Gondor  |  8  |  0x20  |  0xA1  |  0x5E  |  0xEA  |
|  Reflections  |  9  |  -  |  -  |  0x82  |  0xDE  |
|  [[Mount Doom]]  |  10  |  0x68  |  0x89  |  0xA6  |  0xD2  |
|  Shadows  |  11  |  0x8C  |  0x7D  |  0xCA  |  0xC6  |
|  [[Black Rider]]  |  12  |  0xB0  |  0x71  |  0xEE  |  0xBA  |
|  Bloodlines  |  13  |  0xD4  |  0x65  |  0x12  |  0xAF  |
|  Expanded Middle-earth  |  14  |  -  |  -  |  0x36  |  0xA3  |
|  The Hunters  |  15  |  0x1E  |  0x4E  |  0x5A  |  0x97  |
|  The Wraith Collection  |  16  |  -  |  -  |  0x7E  |  0x8B  |
|  Rise of Saruman  |  17  |  0x64  |  0x36  |  0xA2  |  0x7F  |
|  [[Treachery & Deceit]]  |  18  |  0x88  |  0x2A  |  0xC6  |  0x73  |
|  [[Age's End]]  |  19  |  0xAC  |  0x1E  |  0xEA  |  0x67  |
</WRAP>

There's a couple of things that jump out from the data so arranged.  First, the Sequence Bytes get steadily larger from each set to the next.  This suggests that the game client kept all the cards in a giant list, and so the Sequence Byte represented an offset to this giant table, with the Repeating Byte further narrowing it down.  In that case it also seems that the foil sets would fit interlockingly between the normal sets, which would be a mess if the list ever needed manual modification, but isn't really a problem when you're using a table lookup anyway.

This theory is torn apart a bit by the fact that the Sequence Byte appears to reset right around //Return of the King//, which of course would mean that two different sets could map to the same location on the table, which is trouble.  That could be where the !Set Byte is used; if the !Set Byte is smaller than 7, map to table 1, else map to table 2.  This is pure speculation, however.

The Sequence Byte reset happens right on top of Anduril, Flame of the West, and results in 16 cards that have a Sequence Byte of ''0x00''.  Besides this causing problems with parsing, this also makes the identifier byte change halfway through a set; as far as I have been able to determine, this is the only place that happens in the entire collection.

(This is also the entire reason I opted to not consider the null termination at the end of each card as a part of the card's ID; if you rely on null termination to divide the list at all, you end up cutting those 16 cards in half, which eventually blows up on you.  I really have no clue why the developers included //any// null padding in the entire file format; it doesn't align things up to any obvious pattern, and where it might turn out useful it just causes problems like this.  It'd cut file size by about 20% if they were removed, too.)

At any rate, how Worlds Apart organized their information is only of passing interest, as interpreting the data as we have it is most important.  The set name can be determined simply by means of lookup table against the Identifier Byte as shown above.  The card number itself can be determined through a variation of the length format formula:

  repeatingByte + (16 * (sequenceByte - multiplier))
  
The only difference between this and the length format formula is that the multiplier varies from set to set.  While attempting to reverse engineer it, I attempted to come up with a cohesive formula for determining each multiplier, but in the end with so few multipliers actually needed, it turned out to be easier to just generate a deck with card 1 of each set, and adjust the multiplier variable until the calculated value matched the actual card number.  

The multipliers for each set are below, in decimal:

<WRAP indextab>
^  Regular !Sets  ^  Multiplier  ^
|  Promotional (2-byte)  |  117  |
|  Fellowship of the Ring  |  40  |
|  Mines of Moria  |  78  |
|  Realms of the Elf-Lords  |  114  |
|  The Two Towers  |  150  |
|  Battle of Helm's Deep  |  186  |
|  Ents of Fangorn  |  222  |
|  Return of the King 1  |  258  |
|  Return of the King 2  |  2  |
|  Siege of Gondor  |  38  |
|  Reflections  |  136  |
|  [[Mount Doom]]  |  110  |
|  Shadows  |  146  |
|  [[Black Rider]]  |  182  |
|  Bloodlines  |  218  |
|  Expanded Middle-Earth  |  60  |
|  The Hunters  |  34  |
|  The Wraith Collection  |  132  |
|  Rise of Saruman  |  106  |
|  Treachery & Deceit  |  142  |
|  Age's End  |  178  |

^  Foil !Sets  ^  Multiplier  ^
|  Promotional (3-byte)  |  66  |
|  Fellowship of the Ring  |  104  |
|  Mines of Moria  |  140  |
|  Realms of the Elf-Lords  |  176  |
|  The Two Towers  |  212  |
|  Battle of Helm's Deep  |  248  |
|  Ents of Fangorn  |  28  |
|  Return of the King  |  64  |
|  Siege of Gondor  |  100  |
|  [[Mount Doom]]  |  172  |
|  Shadows  |  208  |
|  [[Black Rider]]  |  244  |
|  Bloodlines  |  24  |
|  The Hunters  |  96  |  
|  Rise of Saruman  |  168  |
|  Treachery & Deceit  |  204  |
|  Age's End  |  240  |
|  D-series   |  22  |
|  W-series 1   |  110  |
|  W-series 2   |  172  |
|  Tengwar  |  117  |
</WRAP>

All of them are even except for the Promotional sets, which can be explained by the fact that the Promotional set has one byte's worth of cards that use the 1-byte system and that byte needs to be subtracted from the multiplier.  


As an example, here is a mystery card:

  6c 82 de 08
  
Looking at the Identifying Byte in the third position, we can reference the tables above to see that this card is from the Reflections set (we can also follow the convoluted history of the Set Byte to see this, but in general I've found the Identifier Byte to be more useable).  Combining the Repeating Byte and Sequence Bytes together into the formula given above, along with the multiplier that we know Reflections uses, we get (all numbers converted to decimal):

  108 + (16 * (130 - 136))

This gives us a card number of 12.  The card is set 9, card 12: Aiglos.  


===Tengwar and D-Series===
The formulas and systems discussed above allow us to determine all regular and foil cards, as they follow a sane sequence.  Tengwar cards, however, are all shunted into the same fake !set in spite of spanning several physical sets.  D-series cards (and, presumably, W-series if they were exposed in the client) also are part of a set that is not reflected in the !master database on this site.  Therefore, both of these types of cards need to be given special treatment to accurately determine their card number.

In both cases, there's simply nothing to be done but to have a lookup table that the fake set and card numbers line up to.  I have taken the liberty of compiling the values for the Tengwar cards below:

<WRAP indextab>
^  Fake #  ^  Actual Card  ^  Fake #  ^  Actual Card  ^
|  1  |  01001T  |  29  |  04090T  |
|  2  |  01013T  |  30  |  04100T  |
|  3  |  01014T  |  31  |  06088T  |
|  4  |  01030T  |  32  |  04219T  |
|  5  |  01050T  |  33  |  04176T  |
|  6  |  01072T  |  34  |  04225T  |
|  7  |  01083T  |  35  |  04019T  |
|  8  |  01089T  |  36  |  04289T  |
|  9  |  01096T  |  37  |  05116T  |
|  10  |  01114T  |  38  |  07002T  |
|  11  |  01127T  |  39  |  07211T  |
|  12  |  01165T  |  40  |  07221T  |
|  13  |  01231T  |  41  |  07227T  |
|  14  |  01237T  |  42  |  07321T  |
|  15  |  01256T  |  43  |  07324T  |
|  16  |  02052T  |  44  |  08015T  |
|  17  |  02102T  |  45  |  08025T  |
|  18  |  02105T  |  46  |  08038T  |
|  19  |  04001T  |  47  |  08051T  |
|  20  |  04073T  |  48  |  08057T  |
|  21  |  04364T  |  49  |  08103T  |
|  22  |  05029T  |  50  |  10006T  |
|  23  |  05025T  |  51  |  10009T  |
|  24  |  04154T  |  52  |  10025T  |
|  25  |  04173T  |  53  |  10088T  |
|  26  |  04301T  |  54  |  10122T  |
|  27  |  05100T  |  55  |  07079T  |
|  28  |  04103T  | | |
</WRAP>

In the case of the D-Series card(s), only 1 of them is exposed within the client, and being as they are not used //anywhere// outside of LotR-O, I have left determination of any values for those card(s) as an exercise to the reader.


<sub>Article and research originally by teltura</sub>

{{page>:electronic_platform_table}}