If you’re a data scientist or data engineer, this might sound familiar while working on ETL projects:

• Switching between multiple projects is a hassle
• Debugging others’ code is a nightmare
• Spending a lot of time solving non-business-related issues

SETL (Spark ETL, pronounced “settle”) is a Scala framework that helps you structure your Spark ETL projects, modularize your data transformation logic and speed up your development.

Table of contents

• Quick start
• Setl
• Data Access Layer
  • Access data with Connector
    • FileConnector
    • DBConnector
    • StructuredStreamingConnector
    • Use your own connector
  • Access data with Repository
  • Create a connector
  • Create a repository
  • Hide implicit data conversion with SparkRepositoryAdapter
  • Configuration Example
• Data Transformation API
  • Transformer
  • Factory
• Workflow Management
  • Stage
  • Pipeline
  • Pipeline execution optimization (preliminary feature)
• Utilities
  • Annotations
  • SparkSession builder
  • ConfigLoader
  • DateUtils
  • Condition
• Developer
  • StructAnalyser
  • SchemaConverter
  • PipelineInspector
  • PipelineOptimizer
  • DeliverableDispatcher
  • Read cache strategy
  • Logging