开发者

sometime Not enough replicas available for query at consistency ONE (1 required but only 0 alive) error

when insert date into cassandra, I use spark structured stream kafka-cassandra. data insert no problem. but when i load data. sometime occurred error, SQL Error: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)

my spark-submit code

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,com.github.jnr:jnr-posix:3.1.15 --conf spark.dynamicAllocation.enabled=false --conf com.datastax.spark:spark.cassandra.connectiohost=172.16.10    .6,spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions

source code

# Spark Bridge local to spark_master
spark = SparkSession.builder \
开发者_高级运维    .master(_connSession)\
    .appName("Spark_Streaming+kafka+cassandra") \
    .config('spark.cassandra.connection.host', _connCassandraHost) \
    .config('spark.cassandra.connection.port', _connCassandraPort) \
    .getOrCreate()

# readStream From DATA_SOC at DB Server
df = spark.readStream \
  .format("kafka") \
  .option("kafka.bootstrap.servers", _connKafka) \
  .option('startingOffsets','earliest') \
  .option("subscribe", _topic) \
  .option('failOnDataLoss', False) \
  .load() \
  .select(from_json(col("value").cast("String"), schema).alias("parsed_value")) \
  .select("parsed_value.*")

df = df.withColumn("create_dt",date_format(current_date(),"yyyyMM"))
df = df.withColumn("max",df.attributes.getItem("max"))
df = df.withColumn("min",df.attributes.getItem("min"))
df = df.withColumn("avg",df.attributes.getItem("avg"))
df = df.drop("attributes")

df.printSchema()

# writeStream at cassandra in Spark master VM
ds = df.writeStream \
  .trigger(processingTime='15 seconds') \
  .format("org.apache.spark.sql.cassandra") \
  .option("checkpointLocation","/home/jeju/jeju_sensor_collector/src/checkPoint") \
  .options(table=_table,keyspace=_keySpace) \
  .outputMode('append') \
  .start()

ds.awaitTermination()

application Desigh Pattern is Master-Slave, 1master-3slave.

why auto off spark-cassandra??

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜