OPAL: Leveraging Open Source
dc.contributor.author | Ogden, Eddge | |
dc.contributor.author | Call, Kenneth | |
dc.contributor.author | Myers, Isaac J. | |
dc.contributor.author | Ma, Zayd L. | |
dc.contributor.author | Lowe, Daniel | |
dc.date.accessioned | 2022-11-24T01:32:51Z | |
dc.date.available | 2022-11-24T01:32:51Z | |
dc.date.issued | 2022-10 | |
dc.identifier.citation | Ogden, E., Call, K., Myers, I. J., Ma, Z. L., & Lowe, D. (2022). OPAL: Leveraging Open Source. International Telemetering Conference Proceedings, 57. | |
dc.identifier.issn | 1546-2188 | |
dc.identifier.issn | 0884-5123 | |
dc.identifier.issn | 0074-9079 | |
dc.identifier.uri | http://hdl.handle.net/10150/666979 | |
dc.description.abstract | The current explosion of test and evaluation data being collected from various systems has exposed a strong need for low-cost digital infrastructure to facilitate scalable analytics across all available data. Private industry and academic research have built such systems utilizing Open-Source Software (OSS) with tremendous success. The 309th Software Egineering Group (SWEG) developed OPAL (Open Platform for Advanced Learning) platform is a government owned and developed solution to address this gap and provide data discovery, analytics, and warehousing all license free. OPAL leverages best-in-breed Open-Source Software including JupyterLab (Python analysis environment), MinIO (S3-compliant, redundant and object-versioning data backend), Postgres (Data cataloging), and Dask (scalable compute), among others. In addition to Open-Source tooling, custom integration and software piping are used to further lower the analysts’ barrier to available data: custom Chapter 10 parsing and translating at high speed (10GB/min) into Apache Parquet format, a web-based data catalog for discovery, and lightweight arbitrary object storage organization. This paper will delineate design choices, our DevOps paradigm, benchmarking numbers, and results against a publicly available commercial flight dataset. | |
dc.description.sponsorship | International Foundation for Telemetering | |
dc.language.iso | en | |
dc.publisher | International Foundation for Telemetering | |
dc.relation.url | http://www.telemetry.org/ | |
dc.rights | Copyright © held by the author; distribution rights International Foundation for Telemetering | |
dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | |
dc.title | OPAL: Leveraging Open Source | |
dc.type | Proceedings | |
dc.type | text | |
dc.contributor.department | 309th SWEG | |
dc.contributor.department | Air Force Materiel Command Hill AFB, UT | |
dc.contributor.department | Avionics Test and Analysis Corporation | |
dc.identifier.journal | International Telemetering Conference Proceedings | |
dc.description.collectioninformation | Proceedings from the International Telemetering Conference are made available by the International Foundation for Telemetering and the University of Arizona Libraries. Visit https://telemetry.org/contact-us/ if you have questions about items in this collection. | |
dc.eprint.version | Final published version | |
dc.source.journaltitle | International Telemetering Conference Proceedings | |
refterms.dateFOA | 2022-11-24T01:32:52Z |