Comparing Web Archives: EOT2008 and EOT2012 – What

This post carries on from where the previous post in this series ended.

A very quick recap,  this series is trying to better understand the EOT2008 and the EOT2012 web archives.  The goal is to see how they are similar, how they are different, and if there is anything that can be learned that will help us with the upcoming EOT2016 project.

What

The CDX files we are using has a column that contains the Media Type (MIME Type) for the different URIs in the WARC files.  A list of the assigned Media Types are available at the International Assigned Numbers Authority (IANA) in their Media Type Registry.

This is a field that is inherently “dirty” for a few reasons.  This field is populated from a field in the WARC Record that comes directly from the web server that responded to the initial request.  Usually these are fairly accurate but there are many times where they are either wrong or at the least confusing.  Often times this is caused by  a server administrator, programmer, or system architect that is trying to be clever,  or just misconfigured something.

I looked at the Media Types for the two EOT collections to see if there are any major differences between what we collected in the two EOT archives.

In the EOT2008 archive there are a total of 831 unique Mime/Media Types,  in the EOT2012 there are a total of 1,208 unique type values.

I took the top 20 Mime/Media Types for each of the archives and pushed them together to see if there was any noticeable change in what we captured between the two archives.  In addition to just the raw counts I also looked at what percentage of the archive a given Media Type represented.  Finally I noted the overall change in those two percentages.

Media Type 2008 Count % of Archive 2012 Count % of Archive % Change Change in % of Archive
text/html 105,592,852 65.9% 116,238,952 59.9% 10.1% -6.0%
image/jpeg 13,667,545 8.5% 24,339,398 12.5% 78.1% 4.0%
image/gif 13,033,116 8.1% 8,408,906 4.3% -35.5% -3.8%
application/pdf 10,281,663 6.4% 7,097,717 3.7% -31.0% -2.8%
4,494,674 2.8% 613,187 0.3% -86.4% -2.5%
text/plain 3,907,202 2.4% 3,899,652 2.0% -0.2% -0.4%
image/png 2,067,480 1.3% 7,356,407 3.8% 255.8% 2.5%
text/css 841,105 0.5% 1,973,508 1.0% 134.6% 0.5%

Because I like pictures here is a chart of the percent change.

Change in Media Type

If we compare the Media Types between the two archives we find that the two archives share 527 Media Types.  The EOT2008 archive has 304 Media Types that aren’t present in EOT2012 and EOT2012 has 681 Media Types that aren’t present in EOT2008.

The ten most frequent Media Types by count found only in the EOT2008 archive are presented below.

Media Type Count
no-type 405,188
text/x-vcal 17,368
.wk1 8,761
x-text/tabular 5,312
application/x-wp 5,158
* 4,318
x-application/pdf 3,660
application/x-gunzip 3,374
image/x-fits 3,340
WINDOWS-1252 2,304

The ten most frequent Media Types by count found only in the EOT2012 archive are presented below.

Media Type Count
warc/revisit 12,190,512
application/http 1,050,895
application/x-mpegURL 23,793
img/jpeg 10,466
audio/x-flac 7,251
application/x-font-ttf 7,015
application/x-font-woff 6,852
application/docx 3,473
font/ttf 3,323
application/calendar 2,419

In the EOT2012 archive the team that captured content had fully moved to the WARC format for storing Web archive content.  The warc/revisit records are records for URLs that had not changed content-wise across more than one crawl.  Instead of storing the URL again, there is a reference to the previously captured content in the warc/revisit record.  That’s why there are so many of these Media types.

Below is a table showing the thirty most changed Media Types that are present in both the EOT2008 and EOT2012 archives.  You can see both the change in overall numbers as well as the percentage change between the two archives.

Media Type EOT2008 EOT2012 Change % Change
image/jpeg 13,667,545 24,339,398 10,671,853 78.1%
text/html 105,592,852 116,238,952 10,646,100 10.1%
image/png 2,067,480 7,356,407 5,288,927 255.8%
image/gif 13,033,116 8,408,906 -4,624,210 -35.5%
4,494,674 613,187 -3,881,487 -86.4%
application/pdf 10,281,663 7,097,717 -3,183,946 -31.0%
application/javascript 39,019 1,511,594 1,472,575 3774.0%
text/css 841,105 1,973,508 1,132,403 134.6%
text/xml 344,748 1,433,159 1,088,411 315.7%
unk 4,326 818,619 814,293 18823.2%
application/rss+xml 64,280 731,253 666,973 1037.6%
application/x-javascript 622,958 1,232,306 609,348 97.8%
application/vnd.ms-excel 734,077 212,605 -521,472 -71.0%
text/javascript 69,340 481,701 412,361 594.7%
video/x-ms-asf 26,978 372,565 345,587 1281.0%
application/msword 563,161 236,716 -326,445 -58.0%
application/x-shockwave-flash 192,018 479,011 286,993 149.5%
application/octet-stream 419,187 191,421 -227,766 -54.3%
application/zip 312,872 92,318 -220,554 -70.5%
application/json 1,268 217,742 216,474 17072.1%
video/x-flv 1,448 180,222 178,774 12346.3%
image/jpg 26,421 172,863 146,442 554.3%
application/postscript 181,795 39,832 -141,963 -78.1%
image/x-icon 45,294 164,673 119,379 263.6%
chemical/x-mopac-input 110,324 1,035 -109,289 -99.1%
application/atom+xml 165,821 269,219 103,398 62.4%
application/xml 145,141 246,857 101,716 70.1%
application/x-cgi 100,813 51 -100,762 -99.9%
audio/mpeg 95,613 179,045 83,432 87.3%
video/mp4 1,887 73,475 71,588 3793.7%

Presented as a set of graphs,  first showing the change in number of instances of a given Media Type between the two archives.

30 Media Types that changed the most

30 Media Types that changed the most

The second graph is the percentage change between the two archives.

% Change in top 30 mimetypes shared between archives

% Change in top 30 media types shared between archives

Things that stand out are the growth of application/javascript between 2008 and 2012,  up 3,774% and application/json that was up over 17,000%.  Two formats used to deliver video grew as well with video/x-flv and video/mp4 increasing 12,346% and 3794% respectively.

There were a number of Media Types that reduced in the number and percentage but they are not as dramatic as those identified above.  Of note is that between 2008 and 2012 there was a decline of 100% in content with a Media Type of application/x-cgi and a 78% decrease in files that were application/postscript.

Working with the Media Types found in large web archives is a bit messy.  While there are standard ways of presenting Media Types to browsers, there are also non-standard, experimental and inaccurate instances of Media Types that will exist in these archives.  It does appear that we can see the introduction of some of the newer technologies between the two different archives.  Technologies such as the adoption of JSON and Javascript based sites as well as new formats of video on the web.

If you have questions or comments about this post,  please let me know via Twitter.