August 14, 2015

QIC-24 Tape Data Block Format Decoding & Analysis

Please also visit QICreader.com

 So far, it seems that the comprehensive details of how the magnetic flux transitions are arranged on QIC tape using the QIC-24 standard, have been lost in the halls of time, and never made it to the internet, at least not in a complete way.

 In my opinion, QIC.org should be the most likely to have this information, but they seem lacking in the QIC-24 (and QIC-11) department.

 Well, I am hoping that this page will fill in a lot of the missing information about QIC-24 tape format.

 Before diving in, I wanted to clarify some of the "standard" terminology that surrounds this very mid-1980s QIC tape technology.  I've tried to simplify the definitions, and I put them in my own words.

 QIC-24 is a tape data read/write format, defined best by 

And here is another similar diagram



 QIC-11 is a very similar, yet older, data read/write format, defined best by the below diagram.  The main difference seems to be the block address size (1 byte vs 4 bytes on QIC-24).
For this web page, we are focusing only on QIC-24 format, with the hopes to eventually also analyze the older QIC-11 format later. 
QIC-36 is a hardware interface, consisting of a 50-pin connector, where 25 pins are utilized, and 25 pins are ground.  The detailed description & pinouts can be found in section 4 of the:

 QIC-02 is a hardware interface, and is the interface that the MightyFrame motherboard (as well as many other vintage mini-micro systems from that era) uses to talk to the QIC tape drive assembly.  

 Therefore, a "formatter/controller" board must be present in between the motherboard and the QIC-36 interface of the actual tape drive.  In the case of both the Archive 5945 and the Wangtek 5099, this controller board sits in the same "full-height" chassis as the tape drive, directly beneath it.  The analysis done on this page will focus on the QIC-36 interface, and just let the QIC-02 interface do what it does.   
Computer Archetecture and Organization, by B. Govindarajalu, Page 443

Please note, some have suggested that there is a SCSI interface involved here, but my tests have shown that the pinout for any of the interfaces do not match SCSI, even though they both use 50-pin connectors.  Some formatter boards do interface SCSI to QIC-36, but not the ones I'm working with on this MightyFrame model.  Instead, my tests have determined that I am dealing with QIC-36 and QIC-02 interfaces in this vintage of MightyFrame system.

 My Preview

 So, in order to read the data, I connected a Saleae Logic Analyzer to these pins on my custom-built QIC-36 diagnostic light indicator box.

2 (Go) 
4 (Reverse) 
26 (Read Data Pulse)
38 (Tachometer)

 Here is a YouTube demonstration of reading a file with this custom-built QIC-36 diagnostic light indicator box.


And later, I've discovered that it is more beneficial for me to collect less information on the logic analyzer, so I reduced the signals I'm capturing to only:

26 (Read Data Pulse)

Here's a 25 minute demo of the full tape data pulse capture process and mechanisms:

So, now, let's dive deep into the anatomy of the QIC-24 data block format:

Preamble
From Computer Peripherals, Chapter 11: Magnetic Tape Drives, page 7 and
Computer Archetecture and Organization, by B. Govindarajalu, Page 443
Digital Baseband Transmission and Recording by Jan Bergmans, Page 179
Cipher Series 540 Cartridge Tape Drive Product Description, Page 13

 “This is a sequence of bytes used to synchronize the PLL (Phase Locked Loop) to the data frequency.”

So, instead of using the Tachometer signal as a clock to synchronize the PLL for data reading, evidently the preamble does this.

 And, according to the bottom of page 13 of the Cipher manual linked above:

The preamble is also used to measure the average [data pulse?] amplitude.  There are three preamble lengths, measured by a range of flux transitions per inch (fti).  These preamble lengths are:

  • Normal (120 to 300 flux transitions)
  • Elongated (3,500 to 7,000 flux transitions).  This preamble preceeds the first data block recorded after an underrun.

  • Long (15,000 to 30,000 flux transitions).  This preamble precedes the first data block for interchange, recorded at the beginning of track 0.

 GCR 4/5 Encoding
From Computer Peripherals, Chapter 11: Magnetic Tape Drives, page 7 and
Computer Archetecture and Organization, by B. Govindarajalu, Page 443
Digital Baseband Transmission and Recording by Jan Bergmans, Page 179

QIC-24 format evidently uses GCR 4/5 encoding....so what does that mean?

There seems to be more than one type of GCR 4/5 encoding.  Commodore used one, but it didn't fit here.  Instead, I found the source which gives a table for GCR code used in a "variety of magnetic tape drives".

And, look at this!  I just found it again in the Archive Scorpion 1/4" Streaming Tape Drive Product Description - March 1984, Page 5-4.  Only there they call it 4-to-5 Run Length Limited (RLL).

But, BONUS, this shows the code for the Filemark (more on that below).  00101


Data Block Marker
From Computer Peripherals, Chapter 11: Magnetic Tape Drives, page 7 and
Computer Archetecture and Organization, by B. Govindarajalu, Page 443

GCR code '11111 00111'

Note that NEITHER of these 5-digit codes exist in the GCR 4/5 table above?  That's good to know that they're using unique characters.  As you see below, however, the separation between the end of the preamble and the beginning of this marker was tricky to identify.


Data Block

The 512 data bytes (never more, never less) encoded as GCR 4/5.


Block Address

The block address consists of a 4-Byte block address with track number, a control nibble and block number starting with 1 on each track. 

Control Nybble
The text pictured below refers to the Control nibble within the Block Address, shown in the diagram above.
From AMERICAN NATIONAL STANDARD X3.136-1986, Page 16 (no online version available)


Block Number
The text pictured below refers to the Block Number within the Block Address, shown in the diagram above.
From AMERICAN NATIONAL STANDARD X3.136-1986, Page 16 (no online version available)



Cyclic Redundancy Check

A 2-Byte CRC character is calculated over the 512 bytes of data and 4 bytes of block address using the following polynomial:

CRC = x16 + x12 + x5 + 1

So, I looked up some pages on how to use this polynomial to calculate the CRC, but I found it VERY confusing.  It's not as simple as straight-forward algebra.  I will likely dedicate a whole new page to an explanation on how to calculate this once I figure it out and test it.

 Chuck(G) on vintage-computer.com suggests using http://reveng.sourceforge.net/

 11/15/2015 update:

 I tried my hand at calculating this, once I learned more about how CRC works, and that the polynomial I list above is equivalent to the binary:  10001000000100001

 But, simple online calculators weren't giving me the result I expected.  Then, I thought I saw a clue in this, although I didn't know what it meant:
From AMERICAN NATIONAL STANDARD X3.136-1986, Page 16 (no online version available)

 "The cyclical redundancy check (CRC) shall consist of two bytes, calculated
over the 512 bytes of interchange data, and the 4-byte block address, starting
with all ONEs, CRC initial value, and using the CRC Generating polynomial:
x16 + x12 + x5 + 1"

 *"starting with all ONEs, CRC initial value"*?  What does THAT mean?  

 Lawrence at classiccmp.org, among other contributors, helped me find a work-around, even though I'm not sure I understand it as of yet:

 Prepending 84CF to the 512-byte data block and the 4-byte block address.  It works!

According to Lawrence:  It's the initial value of each bit, corresponding to the "Init before
calculation" lines in the code at the bottom of the page. There doesn't
seem to be any provision for entering this value in the online
computation (and all-zeros is *not* a good initialisation vector!)

 One way around this would be to come up with a byte sequence that
gave a CRC of all ones (i.e. 84 CF) and pre-pend that to the main
sequence, and then it works. 

 Thanks, Lawrence!  Now, for a simple online check, if I use this online calculator, and paste the data block + block address after the 84CF, I get the expected CRC.
 https://ghsi.de/CRC/index.php?Polynom=10001000000100001&Message=84CF

Thanks, Lawrence, Dwight, Chuck, and all who helped me understand how this works at classiccmp.org.


A normal post amble of 5 - 20 flux transitions are recorded at maximum density to act as a guard band. Elongated post ambles of 3500 - 7000 flux transitions are appended when an underrun occurs.

 This means that the Postamble is significantly shorter than the preamble, especially the preamble before the first data block. 
Data Pulse Capture & Analysis

Now to the really fun stuff...I'm using Saleae Logic Analyzer 1.1.15, which I downloaded for free from
https://www.saleae.com/downloads 

 In order to make finding the data more simple, I created a series of 2-block (1024 byte) files, most containing a repetition of a single hexadecimal value.  For testing, I used these HEX values:

 00, 01, 02, 04, 08, 20, 40, FF

 I also had a mixed character/value source file, which I called "Sample1".  

 I then wrote these files to a DC600 tape with a Tandberg TDC-3620 connected to a machine running modern Linux.  

 Then I did the Saleae Logic capture on each file, reading the tape with a Wangtek 5099, connected via the custom box I show in one of the videos above, to an Archive brand QIC-36 to QIC-02 "formatter/controller" board, which in turn is connected to the MightyFrame motherboard.

Analysis Source File Downloads

I am providing downloads of all of the source files, as well as the .logicdata capture files from my Saleae Logic Analyzer on this Google Drive Here.


So, armed with this information, and after much study, trial-and-error, and meeting with many VERY smart people who had some concept about how data was commonly read from magnetic flux transition on other 1980s computer peripherals, I was able to scroll through the Saleae Logic Analyzer capture file, and find the first block of data. 
QIC-Hex-00 
(click here for hi-res image)

As soon as I was able to decode this, I started working on the block address and CRC on the end of the data block, but then I realized, how will I know where the zeros from this file end, and any zeros from the block address begin?  Well, I just moved onto a file that was as far from all zeros as I could get:


And now it all becomes clear.  Again, knowing exactly where these nybbles begin and end, and notice, once again, how the last one extends into what appears to be the postamble.

 So, just to make sure I had the codes right, I did analysis in the beginning of my "Sequence1" file, where I put ascending hex characters at the beginning.

Sequence1
(click here for hi-res image)

 Then one quarter way through (half way through the first block), I put in alpha-numeric characters, starting with the alphabet lower-case.  That's all the further I felt I needed to go to verify that I had this right.


Filemarks

 And, since I discovered that filemarks written by these tape drives were special codes that I could not replicate in data byte sets, I've always wondered what they looked like.  Now I know.

 A filemark is a regular, yet special, 512-byte data block.  Notice how the nybbles found within the data block here are also, NOT found on the GCR 4/5 table.  That's what makes them special, oh so special.
Filemark

And then I found this, which gives the most thorough definition of the "rules" of the QIC-24 filemarks that I've ever seen:  File Mark Block. 

The file mark block format shall be identical to the data block format except that the data field shall contain 512 bytes consisting of the following GCR pattern:  00101  (see excerpt pictured below) 
From AMERICAN NATIONAL STANDARD X3.136-1986, Page 16 (no online version available)

And, it goes on to say "The postamble of a file mark block shall always be an elongated postamble."  Good to know.

 "The GCR nibble 00101 shall be converted to the data nibble 1111 to form the data field for CRC generation and checking."  Even better to know, for when we test the CRC formulas.

I still need to verify this, but I think that, if the data block prior to the filemark is the last one written on the tape (for the last recording session at least, not to be confused with any data previously on the tape but not erased after this point), that the MightyFrame system, as well as my modern Linux setup, writes 2 filemarks after each file, but then goes back and overwrites one of them if an additional file is written, ensuring that 2 consecutive filemarks indicates the end of data on the tape, while only one filemark indicates that the following block, since it is not a second filemark, begins the next file.

Your Analysis & Critiques

In order to more quickly find the beginning of the data block using my .logicdata files, you may need to follow a procedure similar to this:

Thanks for reading this!  I welcome your feedback, comments, your own analysis findings, or whatever else you have to say about all of this craziness.

 Why?!

 Why would I do all of this analysis?  Especially for 35-year old way obsolete data storage?

 Well, I have tapes that are both QIC-24 and QIC-11 that error out when read.  But why?  I mean, the system doesn't tell me that it's because the block is too short, or the CRC doesn't add up, or that 35% of the flux transitions are blank, or anything, and I want to know.  Otherwise, what hope would I have of ever diagnosing and repairing any of the data on those tapes?

 Also, nobody has ever put a comprehensive guide to reading this vintage of tape to this level of granularity, on the web.  So, it's my way of giving back to the vintage computing world and historical archives.

 I know it's crazy, but everybody's gotta have a hobby (or at least they should...)

 Follow Ups

 Thanks to all of you who posted comments on this blog below.

 In addition:

 Tom Trebisky decided to write a related page on his site at
http://cholla.mmto.org/computers/qic/

 Please also see my posts about this topic at:

 vintage-computer.com thread

 MARCH Yahoo Group thread

 comp.sys.3b1 thread

 ------------------------------------------------------------------

 Would acquiring these documents help?

 ANSI X3.55 1977 1/4-inch wide tape cartridge

ANSI X3.54 1977 1/4-inch wide tape cartridge

They are mentioned here:

 http://bitsavers.trailing-edge.com/pdf/wangtek/200363_5000Eoem_Nov84.pdf#page=6
at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)