Definition

ConnectorBuilder provides a simplified way to create Connector.

Usage

You have two ways to instantiate a ConnectorBuilder:

• with a Typesafe Config object from a configuration file
• with a Conf object from a Map[String, String].

With Typesafe Config

Firstly, you should create a configuration file in your project’s resources directory.

In this case, let’s call it application.conf.

csvConfiguration {
  storage = "CSV"
  path = "your/path/to/file.csv"
  inferSchema = "true"
  delimiter = ";"
  header = "true"
  saveMode = "Append"
}

Then you can use ConfigLoader to load your configuration file. By default, it loads application.conf.

object Properties extends ConfigLoader

val connector = new ConnectorBuilder(spark, Properties.getConfig("csvConfiguration")).getOrCreate()

connector.read()
connector.write(df)

With Conf

You can create a Conf object from a Map.

val conf = Conf.fromMap(
  Map(
    "storage" -> "PARQUET",
    "path" -> "path/to/your/file",
    ...
  )
)

val connector = new ConnectorBuilder(spark, conf).getOrCreate()

connector.read()
connector.write(df)

Parameters

Please refer to Connector documentation