LANDER:internet outage survey it50j-20121027 From Predict README version: 4239, last modified: 2014-08-6. This file describes the trace dataset "internet_outage_survey_it50j-20121027" provided by the LANDER project. [IMG] Warning: This datasets was packaged with additional data. See a full description in User Annotations Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Data Format • 3.1 Legacy Format • 4 Collection Method • 4.1 Probing Location(s) • 4.2 Beginning/Ending Date and Time Zone • 5 Results Using This Dataset • 6 Citation • 7 User Annotations • 7.1 Additional Data Packaged With Dataset LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ internet_outage_survey_it50j-20121027 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ outages in a sample of IPv4 /24 blocks │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ To collect this data, a subset of Internet IP address blocks were pinged by │ │ │ sending ICMP ECHO_REQUEST (PING) packet. The response (if it ever came within 11 │ │ │ minutes time interval) was recorded in this data set. Probe was repeated every 11 │ │ │ minutes. In all, approximately 20k /24 subnets were periodically reprobed. Outages │ │ │ are analyzed and reported on the responses. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2012-10-27 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 00:27:44 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2012-11-10 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 00:28:16 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2014-08-08 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 20:32:26 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:address-space-status-data, subcategory:network-outage, ip-address, sweep, │ │ │ address-collection, ping, icmp │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ text, binary │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 17825792 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 45738830 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/109/1353962 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See https://ant.isi.edu/datasets/#getting-datasets for information on obtaining │ │ │ this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents internet_outage_survey_it50j-20121027.README.txt      copy of this README outage.fsdb.bz2      outage information for each block outages.png graphical representation of outage information for a survey Data Format Data is in FSDB format: #fsdb -F t block start duration uncertainty status Where fields are: block block address of the /24 in hex (with trailing zeros) start when the status was takes effect (seconds since the Unix epoch) duration how long the status is in effect uncertainty our confidence in the precision of the start time. In non-raw data uncertainty is sometimes lowered when we merge observeations from multiple observers. status vantage point of the corresponding survey (each letter 'c','j','w' is one of the sites from our observers; corresponding capital letter 'W', 'C', 'J' means the vantage point saw an outage Legacy Format (if outages-legacy.fsdb.bz2 file is present) hexip ip no_data num_rounds density outages hexip: hex format of /24 IP block ip: decimal format of /24 IP block no_data: rounds where we have no useful data (usually when there is an local outage) num_rounds: number of 11-minute rounds in this survey, for this block density: average response rate of this block outages: the rounds that we detect outages, in the format of a-b;c-d. (a,b,c,d are all rounds) Collection Method Data collection involves pinging a fraction of allocated address blocks. Full description of this method is in: > Lin Quan, John Heidemann and Yuri Pradkin. Detecting Internet Outages with Precise Active Probing (extended). > ISI-TR-2012-678. February, 2012 http://www.isi.edu/~johnh/PAPERS/Quan12a.html. This specific work involved probing all addresses in each of about 20k /24 blocks for about 2 weeks. Using icmptrain, the tool described in: > John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. > Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. > Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. Probing Location(s) The probing locations for outage surveys are indicated in their names. A "w" means west, which is from isi.edu in Marina del Rey, California; "c" means center, from colostate.edu, in Ft. Collins, Colorado; "e" means east which is from east.isi.edu, from Arlington, Virginia. See Dataset Contents for more information on the system who did the survey. Beginning/Ending Date and Time Zone Dates/Times specified in the metadata are in UTC. Results Using This Dataset Datasets of outages detected in ISI Internet survey data, similar to this one, have been used the following previously published work: • John Heidemann, Lin Quan, and Yuri Pradkin. A Preliminary Analysis of Network Outages During Hurricane Sandy. Technical Report ISI-TR-2008-685b, USC/Information Sciences Institute, November, 2012. (Updated Feb. 2013.) . • John Heidemann. Third-Party Measurement of Network Outages in Hurricane Sandy. In Proceedings of the FCC Workshop on Network Resiliency, Brooklyn, New York, USA, February, 2013. . • John Heidemann. Active Probing of Edge Networks: Hurricane Sandy and Beyond. Talk given at FCC Workshop on Network Resiliency. February, 2013. . • John Heidemann. Active Probing of Edge Networks: Outages During Hurricane Sandy. Talk given at NANOG57 as part of panel hosted by James Cowie. February, 2013. . Citation If you use this trace to conduct additional research, please cite it as: Internet Outages dataset, IMPACT ID: USC-LANDER/internet_outage_survey_it50j-20121027/rev4239 . Traces taken 2012-10-27 to 2012-11-10. Provided by the USC/LANDER project (http://www.isi.edu/ant/lander). User Annotations Additional Data Packaged With Dataset This dataset includes some miscellaneous data used in Hurricane Sandy analysis. The additional directory "analysis" contains: margin.US.fsdb.bz2      marginal distribution of outages for each round for the US blocks outages.US.only.city-level.fsdb.bz2      US outages with geo data (country=US, region, city, postal_code, lat, lon) outages.US.only.fsdb.bz2      like main outage.fsdb, but only for US blocks outages.US.only.png      like main outage.png, but only for US blocks outages.US.only.with.row.fsdb.bz2      blocks with the "row" key in "outages.US.only.png" outages.with.country.fsdb.bz2      outages.fsdb augmented with country info select_outages/      a few select outages indexed by round outages-legacy.fsdb.bz2      old legacy format used as a source for outages.fsdb.bz2 converter.sh      converter script (relies on FSDB) used to convert outages-legacy.fsdb to outages.fsdb This data is provided "as-is" in hope that it will be useful for other researchers. Yuri (talk) 10:22, 6 August 2014 (PDT) Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Adaptive Probing • LANDER:Datasets:AddressSpace • LANDER:Datasets:AddressSpace:Production Outage