The Confusatory

iTunes’s iCloud Cache

I had a few hours to kill on the plane on Sunday, so I thought I’d take a look at how iTunes does caching for iCloud songs. This is a new caching mechanism in iTunes 10.5, and only applies to songs you start playing from iCloud in iTunes without hitting the download button. (Downloaded songs are treated like regular store purchases.)

Temporary files from iCloud are stored in ~/Library/Caches/com.apple.iTunes/PlayCache. At the top level of this directory is PlayCacheInfo.xml, a file which describes the total cache size, the account they’re from (they store the DSID, which is the numerical version of your Apple ID), the number of items in the cache, timestamps like last-delete-check, last-load-time, and last-written, a path to the library it’s associated with, and an array of items. Each item contains its access date, its iCloud ID (an int that presumably uniquely identifies this file in iCloud’s music storage system), and its size in bytes. PlayCacheInfo.xml appears to be written out when quitting iTunes.

Files are saved on disk in PlayCache in a fan-out structure which looks like the following:

<DSID in 16-character hexadecimal>/
    <two-digit folder>/
        <two-digit folder>/
            <two-digit folder>/
                <hexadecimal DSID>-<song's iCloud ID in 16-character hexadecimal>.m4a

When a file starts streaming from iCloud, it’s stored in a Downloads folder under the hexadecimal DSID folder. Inside Downloads is a temporary folder named for the file being downloaded (in the rough format of <song name>_<album name>_<artist name>.tmp), and inside that folder is a download.m4a file and an Info.plist containing some metadata about the m4a file. The file appears to not be a valid m4a file and won’t open in Fission or iTunes (though iTunes is able to play this same file as it’s downloading from iCloud).

The Info.plist contains metadata like the DSID (an int) for the iCloud account downloading the file, the expected full file size in bytes (an int), an iCloud ID for the file (an int), a download-file-name (where the downloaded file is being saved on disk) (a string), a downloadID (a string), an id (which is always 0 for me) (an int), a flag for is-cloud-redownload (a boolean), and an MD5 hash (a string). It also contains a dictionary called partial-md5, which contains a key called partial-md5-info (a string) and another key called partial-md5-offset (an int).

partial-md5-offset looks to be a byte offset for doing a rolling MD5 hash of the file. It’s a 320 byte string with some constant but non-zero padding in the first 26 bytes, followed by some md5-ish hashes. It initially starts with 26 bytes of padding, a 40 byte hash, and zero padding to the remainder of the string when the file is a placeholder file that iTunes can’t start playing. After the file is more fully downloaded, the rest of the string after the first 26 bytes contains a series of hashes, and the offset increases towards the end of the file. I can’t figure out where these partial or full-ish hashes are coming from though, as their byte offset perfectly matches the file’s current size in bytes, and the hash of the file at that size doesn’t match any of the stored hashes in the string. Maybe it’s the hash for the next incoming chunk? Also interesting is the md5 key: the value of it doesn’t match the fully-downloaded file’s hash.

Open questions:

• How does iTunes use and/or calculate the partial hashes?
• What is stored in partial-md5-info? What’s the 26 byte padding, and what’s the rest of it?
• Why doesn’t md5 match the actual MD5 hash for the fully-downloaded file?
• What’s the naming rules for the fanout directory structure? (I’ve only downloaded a few files and it’s hard to suss out thus far. I’m guessing it’s somehow based on the song’s iCloud ID, since PlayCacheInfo.xml doesn’t exist when you first start downloading these files and contains no directory path information for cached files once it is written out, and the directory paths for the cached files don’t change after deleting the cache and recaching the files.)