This course covers Scala, Spark programming, Spark architecture, Spark Streaming, and integration of Spark with the Hadoop ecosystem for developing Big Data applications. In addition, it covers the technologies that integrate well with Spark in the creation of Big Data applications. Students are required to complete weekly reading and/or programming assignments and demonstrate mastery of course topics by developing a final project using Scala, Spark, and complementary Hadoop tools.
Successful completion of the Realtime and Big Data Analytics course is a prerequisite for registering for this course. In addition, students should have: strong programming skills in Java, Python, or C++; experience using Hadoop; coursework in operating systems, networking, and algorithms; familiarity with Linux commands.
In this course we study the craft of software engineering. The semester begins with a thorough review of the Java Language and Virtual Machine, emphasizing library components and language features most relevant to most development projects. Later lectures cover more sophisticated features in the language like concurrency and thread-safety, writing tests, properly designing classes and Object-Oriented programs, and design patterns. Certain non-technical topics are critical to successful software development and are reinforced throughout the semester. These include writing readable code, the long-term consequences of bugs, the psychology of human motivation, successfully developing and reinforcing habits, what science knows about human productivity and drive, and how to motivate ourselves and our team. Lessons are reinforced with weekly assignments that are meticulously graded. Some submissions are critiqued, line-by-line, in classroom code reviews. The end of the semester includes a lecture on how to best prepare for interviews and write an effective resume, incorporating techniques from world-class athletes and performers on how to optimally perform under extreme stress.
This is a capstone course that connects students directly with real-world information technology problems. The goal of this course is to teach the skills needed for success in real-world information technology via a combination of classroom lectures and practical experience with large projects that have been specified by local “clients.” The typical clients are primarily companies, but can also be government agencies or nonprofit organizations. Each project lasts for the entire semester and is designed to involve the full software project life cycle. Examples of such projects are development of software to solve a business problem, including specifying requirements, writing and testing prototype code, and writing a final report; and evaluation of commercial software to be purchased to address a business problem, including gathering requirements, designing an architecture to connect the new software with existing systems, and assessing the suitability of available software products.