LANDER:NCCDC logs zeek-20170413 From Predict README version: 10749, last modified: 2019-10-30. This file describes the trace dataset "NCCDC_logs_zeek-20170413" provided by the LANDER project. Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Dataset Generation • 3.1 Background • 4 Citation • 5 Results Using This Dataset • 6 User Annotations LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ NCCDC_logs_zeek-20170413 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ 2017 NC Cyber Defense Competition, Zeek output │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ This dataset is Zeek output from packet captures from the 2017 National Collegiate │ │ │ Cyber Defense Competition (nccdc.org). CCDC is a multi-day competition that │ │ │ specifically focuses on the operational aspects of managing and protecting an │ │ │ existing commercial" network infrastructure. Teams of undergraduate/graduate │ │ │ students are provided with a fully functional (but insecure) small business │ │ │ network they must secure; maintain; and defend against a live Red Team. Teams must │ │ │ also respond to business tasks called "injects" throughout the competition. Zeek │ │ │ is a popular network monitoring tool. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2017-04-13 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2017-04-15 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2019-11-11 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:synthetically-generated-data, subcategory:experimental-data, synthetic │ │ │ data, nccdc, zeek │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ csv │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 42308993024 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 279818144724 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/115/1364431 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See https://ant.isi.edu/datasets/#getting-datasets for information on obtaining │ │ │ this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents NCCDC_logs_zeek-20170413.README.txt     copy of this README dayone/     nccdc_2017_dayone_zeek_csv_logs/ zeek output in csv format from the first day of competition         conn.log.xz, conn_long.log.xz, etc. specific logs     nccdc_2017_dayone_zeek_extracted_files.zip 73,647 files extracted by zeek from the first day of competition     nccdc_2017_dayone_zeek_json_logs/ zeek output in JSON format from the first day of competition     nccdc_2017_daytwo_zeek_csv_logs/ zeek output in csv format from the first day of competition     nccdc_2017_daytwo_zeek_json_logs/ zeek output in JSON format from the first day of competition     .sha1sum SHA-1 checksum daytwo/     daytwo.NNN.pcap.xz sequentially numbered pcap traces from the second day of competition     .sha1sum SHA-1 checksum The file ".sha1sum" contains SHA1 checksums of individual compressed files. The integrity of the distribution thus can be checked by independently calculating SHA1 sums of files and comparing them with those listed in the file. If you have the sha1sum utility installed on your system, you can do that by executing: sha1sum --check .sha1sum This has to be done before files are uncompressed. Dataset Generation Background This dataset contains Zeek (Bro) output from the pcaps processed in the 2017 National Collegiate Cyber Defense Competition (NCCDC) held in April in San Antonio, TX (http://www.nccdc.org), as present in the dataset LANDER:NCCDC_logs-20170413 That dataset contained packet captures from a simulated network attack and defense scenario. Please see that dataset for details about the scenario, network topology, capture mechanism. Zeek processing was done by Benjamin Bornholm from RIT as part of his Masters Degree. He says: I generated CSV logs, JSON logs, and extracted all known file types. The CSV logs zipped up is 22GBs, the JSON logs zipped up is 25GBs, and the extracted files are 8GBs zipped up. If you would prefer the logs in raw text (no compression) it will be about ~200Gbs of space per format (JSON or CSV), ~400GBs for both format To convert the data set I setup Zeek to listen on a dummy Linux network interface and then used tcpdreplay to pipe data to that interface. This allowed me to play the PCAP in real time to preserve PCAP timestamps. I also used some Zeek scripts from the internet and the ZIPs should contain a Zeek_script_loaded.log (or similar name) to indicate which Zeek scripts I used. A typical zeek command line (from daytwo csv): zeek -i dummy0 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p bro local.zeek zeekctl zeekctl/standalone zeekctl/auto Citation If you use this trace to conduct additional research, please cite it as: NCCDC Logs Zeek, IMPACT ID: USC-LANDER/NCCDC_logs_zeek-20170413/rev10749 . Traces taken 2017-04-13 to 2017-04-25. Traces provided by the Center for Infrastructure Assurance and Security (UTSA/CIAS); Zeek evaluation 2019-08 by Benjamin Bornholm, RIT; data hosted by the USC/LANDER project (http://www.isi.edu/ant/lander). Results Using This Dataset No results yet. User Annotations Currently no annotations. Categories: • LANDER • LANDER:Datasets • LANDER:Datasets:PCH • LANDER:Datasets:NCCDCLogs • Datasets