IncRDD: Incremental Updates for RDD in Apache Spark

Simple item page
dc.contributor.advisorWu, Weili
dc.creatorDodabelle Prakash, Prathish
dc.creator.orcid0000-0003-3516-8316
dc.date.accessioned2017-06-26T11:51:57Z
dc.date.available2017-06-26T11:51:57Z
dc.date.created2017-05
dc.date.issued2017-05
dc.date.submittedMay 2017
dc.date.updated2017-06-26T11:51:57Z
dc.description.abstractData is constantly changing. Today, there can be incremental updates to the existing data. As the data is evolving with new updates, the results of big data applications gradually become out of date and stale. It is required to refresh the results for every update efficiently. Apache Spark is used to process multiple petabytes of data on clusters having thousands of nodes. The core abstraction of Spark is RDD (Resilient Distributed Dataset), which is an immutable collection of elements. Due to the immutability of RDD, Spark works information in parallel, permits information reuse, and handles failures and stragglers productively. But Spark lacks flexibility and efficiency of incremental processing of small updates. In this thesis, IncRDD framework is proposed for incremental processing of updates to the existing data. IncRDD sustains all the powerful features of Spark including parallel processing, data reusability, and fault tolerance. New operations for RDD are implemented to add new records, update the existing records, and delete them. We introduce a new variant of Cuckoo hashing, Dual-CH Fast-Simple. Dual Cuckoo hashing uses two cuckoo hash tables. The first cuckoo table is used to store records, in every partition of a node. The second hash table is used to implement structural sharing, which adds persistence, utilize previous versions, and avoids expensive re-computation. We evaluate IncRDD using incremental algorithms and provide experimental results to show the significant improvement in the performance of Incremental RDD.
dc.format.mimetypeapplication/pdf
dc.identifier.urihttp://hdl.handle.net/10735.1/5417
dc.language.isoen
dc.rightsCopyright ©2017 is held by the author. Digital access to this material is made possible by the Eugene McDermott Library. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author.
dc.subjectBig data
dc.subjectSPARK (Computer program language)
dc.subjectElectronic data processing
dc.subjectParallel processing (Electronic computers)
dc.subjectComputer software—Reusability
dc.subjectFault-tolerant computing
dc.subjectHashing (Computer science)
dc.titleIncRDD: Incremental Updates for RDD in Apache Spark
dc.typeThesis
dc.type.materialtext
thesis.degree.departmentComputer Science
thesis.degree.grantorUniversity of Texas at Dallas
thesis.degree.levelMasters
thesis.degree.nameMSCS

Files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
ETD-5608-7409.37.pdf
Size:
872.59 KB
Format:
Adobe Portable Document Format
Description:
Download

License bundle

Now showing 1 - 2 of 2
No Thumbnail Available
Name:
LICENSE.txt
Size:
1.85 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
5.86 KB
Format:
Plain Text
Description:
Download

Collections

UTD Theses and Dissertations