Retro-Future: The Retrospective Future in the Internet

Project Description

New network security events, from new zero-day attacks to insider threats, are only apparent after they have occurred. By definition, these events work in unexpected ways and only through hindsight may one learns what should have been collected and analyzed. To address these events, one either needs to predict the unexpected or travel back in time to replay the event. The premise of this project is to record enough network state to replay network security events and effectively travel back in time–in effect, to provide an Internet Digital Video Recorder (DVR).

The challenge in this effort is to provide Internet time-travel that is efficient, maximizing the effective history that is saved, to be cost-effective commodity hardware and software, and most importantly, to accommodate permission and privacy constraints that are necessary to deploy this system.

We expect that the resulting Internet DVR will help aid development and testing of network defenses.

Retro-future is a joint research effort of USC Information Sciences Institute Colorado State University’s Network Security lab, and Los Alamos National Laboratory. It is part of the ANT: the Analysis of Network Traffic research group.

This work supported (2012-2016) by Department of Homeland Security Science and Technology Directorate, Cyber Security Division , via SPAWAR Systems Center Pacific under Contract No. N66001-13-C-3001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of SSC-Pacific.

It is also supported (starting in 2016) by the Department of Homeland Security (DHS) Science and Technology Directorate, Cyber Security Division (DHS S&T/CSD) via contract number HHSP233201600010C. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Department of Homeland Security.

People

Manaf Gharaibeh, collaborating faculty (Colorado State University CS Dept.) (currently faculty at Princess Sumaya University For Technology, Jordan; also Colorado State PhD graduate, 2020)
John Heidemann, PI on this project, project leader and professor (USC/ISI)
Christos Papadopoulos, co-PI on this project, professor (University of Memphis) christos.papadopoulos (at) memphis.edu
Yuri Pradkin, researcher (USC/ISI)

Alumni

Calvin Ardi, USC CS PhD graduate (2020) (USC CS Dept. and ISI)
Daniel Byrne, graduate research assistant (LANL)
Xun Fan, USC CS PhD graduate (2015) (USC CS Dept. and ISI)
Paul Ferrell, researcher (LANL)
Gina Fisk, project lead (LANL)
Mike Fisk, co-PI on this project, project leader (LANL)
Andy Gu, undergraduate researcher (USC CS Dept. and ISI)
Zi Hu, USC CS MS graduate (2014) (USC CS Dept. and ISI)
Dan Massey, professor (University of Colorado Boulder CS Dept.)
Neale Pickett, researcher (LANL)
Abdul Qadeer, PhD student (USC CS Dept. and ISI)
Anant Shah, Colorado State U. CS PhD graduate (2018) (Colorado State University CS Dept.)
Shannon (Shane) Steinfadt, researcher and project lead (LANL)
Ben Uphoff, researcher (LANL)
James Wernicke, researcher (LANL)
Han Zhang, Colorado State U. CS PhD graduate (2017) (Colorado State University CS Dept.)

In addition, we thank Dan Massey for his involvement early in the effort. He is currently on leave from Colorado State and this project.

Publications

Lan Wei and John Heidemann 2020. Whac-A-Mole: Six Years of DNS Spoofing. Technical Report arXiv:2011.12978v1. USC/ISI. [PDF] Details
Manaf Gharaibeh, Christos Papadopoulos, John Heidemann and Craig Partridge 2020. Delay-based Identification of Internet Block Movement. Technical Report CS-20-101. Colorado State University Computer Science Department . [PDF] Details
Abdul Qadeer and John Heidemann 2019. Plumb: Efficient Processing of Multi-User Pipelines (Poster). USC/Information Sciences Institute. [PDF] Details
Liang Zhu and John Heidemann 2018. LDplayer: DNS Experimentation at Scale. Proceedings of the ACM Internet Measurement Conference (Boston, Massachusetts, USA, Oct. 2018), to appear. [DOI] [PDF] [Code] Details
Abdul Qadeer and John Heidemann 2018. Plumb: Efficient Processing of Multi-User Pipelines (Poster Abstract). Proceedings of the Symposium on Cloud Computing (Carlsbad, CA, USA, Oct. 2018), 519. [DOI] [PDF] Details
Abdul Qadeer and John Heidemann 2018. Plumb: Efficient Processing of Multi-Users Pipelines (Extended). Technical Report ISI-TR-727. USC/Information Sciences Institute. [PDF] Details
Hang Guo and John Heidemann 2018. IP-Based IoT Device Detection. Proceedings of the ACM SIGCOMM Workshop on IoT Security and Privacy (Budapest, Hungary, Aug. 2018), 36–42. [DOI] [PDF] [Dataset] Details
Calvin Ardi and John Heidemann 2018. Leveraging Controlled Information Sharing for Botnet Activity Detection. Proceedings of the ACM SIGCOMM Workshop on Traffic Measurements for Cybersecurity (Budapest, Hungary, Aug. 2018), 14–20. [DOI] [PDF] Details
John Heidemann 2018. Internet Outages: Reliablity and Security. Invited talk at University of Oregon Cybersecurity Day. [PDF] Details
Hang Guo and John Heidemann 2018. Detecting ICMP Rate Limiting in the Internet. Proceedings of the Passive and Active Measurement Workshop (Berlin, Germany, Mar. 2018), to appear. [PDF] Details
John Heidemann 2018. Outage Clustering: From Leaves to Trees. Talk at CAIDA Active Internet Measurement Workshop (AIMS). [PDF] Details
Basileal Imana, Aleksandra Korolova and John Heidemann 2018. Enumerating Privacy Leaks in DNS Data Collected Above the Recursive. Proceedings of the ISOC NDSS Workshop on DNS Privacy (San Diego, California, USA, Feb. 2018). [PDF] [Dataset] Details
Lan Wei and John Heidemann 2018. Does Anycast Hang up on You (UDP and TCP)? IEEE Transactions on Network and Service Management. 15, 2 (Feb. 2018), 707–717. [PDF] Details
John Heidemann 2018. Internet Reliability, from Addresses to Outages. Talk at MIT CSAIL. [PDF] Details
John Heidemann, Yuri Pradkin and Aqib Nisar 2018. Back Out: End-to-end Inference of Common Points-of-Failure in the Internet (extended). Technical Report ISI-TR-724. USC/Information Sciences Institute. [PDF] Details
Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale. Technical Report 722. USC/Information Sciences Institute. [PDF] [Code] Details
Wouter B. de Vries, Ricardo de O. Schmidt, Wes Hardaker, John Heidemann, Pieter-Tjerk de Boer and Aiko Pras 2017. Verfploeter: Broad and Load-Aware Anycast Mapping. Proceedings of the ACM Internet Measurement Conference (London, UK, 2017), 477–488. [DOI] [PDF] [Dataset] Details
Moritz Müller, Giovane C. M. Moura, Ricardo de O. Schmidt and John Heidemann 2017. Recursives in the Wild: Engineering Authoritative DNS Servers. Proceedings of the ACM Internet Measurement Conference (London, UK, 2017), 489–495. [DOI] [PDF] [Dataset] Details
Kensuke Fukuda, John Heidemann and Abdul Qadeer 2017. Detecting Malicious Activity with DNS Backscatter Over Time. ACM/IEEE Transactions on Networking. 25, 5 (Aug. 2017), 3203–3218. [DOI] [PDF] [Dataset] Details
Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale (abstract with poster). Technical Report ISI-TR-2017-721. USC/Information Sciences Institute. [PDF] [Code] Details
Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale (poster abstract). Proceedings of the SIGCOMM Posters and Demos (Aug. 2017), 60–62. [DOI] [PDF] [Code] Details
Lan Wei and John Heidemann 2017. Does Anycast Hang up on You? IEEE. [DOI] [PDF] Details
Jelena Mirkovic, Genevieve Bartlett, John Heidemann, Hao Shi and Xiyue Deng 2017. Do You See Me Now? Sparsity in Passive Observations of Address Liveness. IEEE International Conference on Traffic Monitoring and Analysis (Dublin, Ireland, Jul. 2017), 1–9. [DOI] [PDF] Details
John Heidemann 2017. Digging in to Ground Truth in Network Measurements. Talk at the Network Traffic Measurement and Analysis PhD School. [PDF] Details
Moritz Müller, Giovane C. M. Moura, Ricardo de O. Schmidt and John Heidemann 2017. Recursives in the Wild: Engineering Authoritative DNS Servers. Technical Report ISI-TR-720. USC/Information Sciences Institute. [PDF] Details
Wouter B. de Vries, Ricardo de O. Schmidt, Wes Hardaker, John Heidemann, Pieter-Tjerk de Boer and Aiko Pras 2017. Verfploeter: Broad and Load-Aware Anycast Mapping. Technical Report ISI-TR-719. USC/Information Sciences Institute. [PDF] [Dataset] Details
Hang Guo and John Heidemann 2017. Detecting ICMP Rate Limiting in the Internet (extended). Technical Report ISI-TR-717. USC/Information Sciences Institute. [PDF] Details
Ricardo de O. Schmidt, John Heidemann and Jan Harm Kuipers 2017. Anycast Latency: How Many Sites Are Enough? Proceedings of the Passive and Active Measurement Workshop (Sydney, Australia, Mar. 2017), to appear. [PDF] Details
John Heidemann 2017. Collecting and Visualizing Outages Over the Long Haul. Talk at CAIDA Active Internet Measurement Workshop (AIMS). [PDF] Details
Lan Wei and John Heidemann 2017. Does Anycast Hang up on You? (extended). Technical Report ISI-TR-716. USC/Information Sciences Institute. [PDF] Details
Anant Shah, Romain Fontugne and Christos Papadopoulos 2016. Towards Characterizing International Routing Detours. Proceedings of the 12th Asian Internet Engineering Conference (AINTEC) (Bangkok, Thailand, Nov. 2016), to appear. Details
Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event. Proceedings of the ACM Internet Measurement Conference (Nov. 2016). [DOI] [PDF] Details
John Heidemann, Ricardo de O. Schmidt and Jan Harm Kuipers 2016. Anycast Latency: How Many Sites are Enough? Presentation at DNS-OARC Meeting. [PDF] Details
John Heidemann, Giovane C. M. Moura, Ricardo de O. Schmidt, and Wouter B. de Vries, Moritz Muller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating Nov. 30. Presentation at DNS-OARC Meeting. [PDF] Details
Jelena Mirkovic, Genevieve Bartlett, John Heidemann, Hao Shi and Xiyue Deng 2016. Do You See Me Now? Sparsity in Passive Observations of Address Liveness (extended). Technical Report ISI-TR-2016-710. USC/Information Sciences Institute. [PDF] Details
Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event (extended). Technical Report ISI-TR-2016-709b. USC/Information Sciences Institute. [PDF] Details
Ricardo de O. Schmidt, John Heidemann and Jan Harm Kuipers 2016. Anycast Latency: How Many Sites Are Enough? Technical Report ISI-TR-2016-708. USC/Information Sciences Institute. [PDF] Details
Z. Hu, L. Zhu, J. Heidemann, A. Mankin, D. Wessels and P. Hoffman 2016. Specification for DNS over Transport Layer Security (TLS) . Technical Report 7858. Internet Request For Comments. [DOI] [PDF] Details
Abdul Qadeer, John Heidemann and Kensuke Fukuda 2016. Improving Long-term Accuracy of DNS Backscatter for Monitoring of Internet-Wide Malicious Activity (poster). Technical Report ISI-TR-2016-707. USC/Information Sciences Institute. [PDF] [Dataset] Details
Manaf Gharaibeh, Han Zhang, Christos Papadopoulos and John Heidemann 2016. Assessing Co-Locality of IP Blocks. Proceedings of the 19th IEEE Global Internet Symposium (San Francisco, CA, USA, Apr. 2016). [PDF] Details
Han Zhang, Manaf Gharaibeh, Spiros Thanasoulas and Christos Papadopoulos 2016. BotDigger: Detecting DGA Bots in a Single Network. Proceedings of the IEEE International Conference on Traffic Monitoring and Analysis (Louvain La Neuve, Belgium, Apr. 2016), 16–21. [DOI] Details
Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2016. T-DNS: Connection-Oriented DNS to Improve Privacy and Security (poster abstract). Technical Report ISI-TR-2016-706. USC/Information Sciences Institute. [PDF] Details
Calvin Ardi and John Heidemann 2016. AuntieTuna: Personalized Content-Based Phishing Detection. Proceedings of the NDSS Workshop on Usable Security (San Diego, California, USA, Feb. 2016). [PDF] [Code] Details
Han Zhang, Manaf Gharaibeh, Spiros Thanasoulas and Christos Papadopoulos 2016. BotDigger: Detecting DGA Bots in a Single Network. Technical Report CS-16-101. Colorado State University . Details
Manaf Gharaibeh, Han Zhang, Christos Papadopoulos and John Heidemann 2015. Assessing Co-Locality of IP Blocks. Technical Report CS-15-103. Colorado State University Department of Computer Science . [PDF] Details
Kensuke Fukuda and John Heidemann 2015. Detecting Malicious Activity with DNS Backscatter. Proceedings of the ACM Internet Measurement Conference (Tokyo, Japan, Oct. 2015), 197–210. [DOI] [PDF] [Dataset] Details
Kensuke Fukuda and John Heidemann 2015. Detecting Malicious Activity with DNS Backscatter (extended). Technical Report ISI-TR-2015-704. USC/Information Sciences Institute. [PDF] [Dataset] Details
Abdulla Alwabel, John Healy, John Heidemann, Brian Luu, Yuri Pradkin and Rasoul Safavian. 2015. Evaluating Externally Visible Outages. Technical Report ISI-TR-701. USC/Information Sciences Institute. [PDF] Details
Calvin Ardi and John Heidemann 2015. Poster: Lightweight Content-based Phishing Detection. Technical Report ISI-TR-2015-698. USC/Information Sciences Institute. [PDF] Details
Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2015. Connection-Oriented DNS to Improve Privacy and Security. Proceedings of the 36thIEEE Symposium on Security and Privacy (San Jose, Californa, USA, May 2015), 171–186. [DOI] [PDF] [Code] [Dataset] Details
Xun Fan, Ethan Katz-Bassett and John Heidemann 2015. Assessing Affinity Between Users and CDN Sites. Proceedings of the 7th IEEE International Workshop on Traffic Monitoring and Analysis (Barcelona, Spain, Apr. 2015). [DOI] [PDF] [Dataset] Details
Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2015. Connection-Oriented DNS to Improve Privacy and Security (extended). Technical Report ISI-TR-2015-695. USC/Information Sciences Institute. [PDF] [Code] Details
Liang Zhu, Zi Hu and John Heidemann 2015. Evaluation of Future DNSSEC Response Sizes at a Root and a TLD Server. [PDF] Details
Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors. Proceedings of the ACM Internet Measurement Conference (Vancouver, BC, Canada, Nov. 2014), 87–100. [DOI] [PDF] Details
John Heidemann 2014. Internet Populations (Good and Bad): Measurement, Estimation, and Correlation. Presentation at ICERM Workshop on Cybersecurity. [PDF] Details
Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2014. T-DNS: Connection-Oriented DNS to Improve Privacy and Security (extended). Technical Report ISI-TR-2014-693. USC/Information Sciences Institute. [PDF] [Code] Details
Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors (extended). Technical Report ISI-TR-2014-691b. USC/Information Sciences Institute. [PDF] Details
Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors (extended). Technical Report ISI-TR-2014-691. USC/Information Sciences Institute. [PDF] Details
Michael E. Fisk and Curtis L. Hash 2014. FileMap: Map-Reduce Program Execution on Loosely-Coupled Distributed Systems. Proceedings of the 4th International Workshop on Cloud Data and Platforms (Amsterdam, the Netherlands, Apr. 2014), to appear. [DOI] [Code] Details
Zi Hu, Liang Zhu, Calvin Ardi, Ethan Katz-Bassett, Harsha V. Madhyastha, John Heidemann and Minlan Yu 2014. The Need for End-to-End Evaluation of Cloud Availability. Proceedings of the Passive and Active Measurement Workshop (Marina del Rey, California, USA, Mar. 2014), 119–130. [DOI] [PDF] Details
Alefiya Hussain, Yuri Pradkin and John Heidemann 2013. Replay of Malicious Traffic in Network Testbeds. Proceedings of the 13th IEEE Conference on Technologies for Homeland Security (HST) (Waltham, Massachusetts, USA, Nov. 2013), (to appear). [PDF] Details
Matt Calder, Xun Fan, Zi Hu, Ethan Katz-Bassett, John Heidemann and Ramesh Govindan 2013. Mapping the Expansion of Google’s Serving Infrastructure. Proceedings of the ACM Internet Measurement Conference (Barcelona, Spain, Oct. 2013), 313–326. [PDF] Details
Matt Calder, Xun Fan, Zi Hu, Ethan Katz-Bassett, John Heidemann and Ramesh Govindan 2013. Mapping the Expansion of Google’s Serving Infrastructure. Technical Report TR 13-935. University of Southern California Computer Science Department. [PDF] Details
Lin Quan, John Heidemann and Yuri Pradkin 2013. Poster Abstract: Towards Active Measurements of Edge Network Outages. Proceedings of the Passive and Active Measurement Workshop (Hong Kong, China, Mar. 2013), 276–279. [DOI] [PDF] Details
John Heidemann 2013. Third-Party Measurement of Network Outages in Hurricane Sandy. Proceedings of the FCC Workshop on Network Resiliency (Brooklyn, New York, USA, Feb. 2013). [PDF] Details

For related publications, please see the ANT publications web page.

Software

antlink Manage a tree of git or other VC repositories with funky symlinks
babarchive Manage babarchives, checksummed directory trees that can be validated
cryptopANT C/C++ Library/tool for IP address anonymization
dag scrubber Tool for scrubbing packet traces
dnsanon extract DNS traffic from pcap to text with optionally anonymization
dnsanon_rssac Dnsanon_rssac is an implementation of RSSAC-002v2 processing for DNS statistics
LANDER Trace Software LANDER Trace Capture software handles for packet capture, scrubbing, and triggering user-provided scripts
LDplayer/dns-replay-client Replay DNS queries against a DNS server with correct timing and optionally log timing or latency.
LDplayer/dns-replay-controller LDplayer component
LDplayer/dns-query-mutator LDplayer component
LDplayer/dns-replay-proxy A proxy that helps to emulate DNS hierarchy in DNS trace replay.
LDplayer/dns-route-setup LDplayer component
LDplayer/dns-zone-constructor LDplayer component
timefind and indexer Software to handle indexing and selection of multiple network data types based on a given time range.

Datasets

The primary goal of Retro-Future is to develop new network measurement capabilities. When we generate new datasets we can release as a side-effect of this process we will announce them here.