sometime Not enough replicas available for query at consistency ONE (1 required but only 0 alive) error
when insert date into cassandra, I use spark structured stream kafka-cassandra. data insert no problem. but when i load data. sometime occurred error, SQL Error: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)
my spark-submit code
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,com.github.jnr:jnr-posix:3.1.15 --conf spark.dynamicAllocation.enabled=false --conf com.datastax.spark:spark.cassandra.connectiohost=172.16.10 .6,spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions
source code
# Spark Bridge local to spark_master
spark = SparkSession.builder \
开发者_高级运维 .master(_connSession)\
.appName("Spark_Streaming+kafka+cassandra") \
.config('spark.cassandra.connection.host', _connCassandraHost) \
.config('spark.cassandra.connection.port', _connCassandraPort) \
.getOrCreate()
# readStream From DATA_SOC at DB Server
df = spark.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", _connKafka) \
.option('startingOffsets','earliest') \
.option("subscribe", _topic) \
.option('failOnDataLoss', False) \
.load() \
.select(from_json(col("value").cast("String"), schema).alias("parsed_value")) \
.select("parsed_value.*")
df = df.withColumn("create_dt",date_format(current_date(),"yyyyMM"))
df = df.withColumn("max",df.attributes.getItem("max"))
df = df.withColumn("min",df.attributes.getItem("min"))
df = df.withColumn("avg",df.attributes.getItem("avg"))
df = df.drop("attributes")
df.printSchema()
# writeStream at cassandra in Spark master VM
ds = df.writeStream \
.trigger(processingTime='15 seconds') \
.format("org.apache.spark.sql.cassandra") \
.option("checkpointLocation","/home/jeju/jeju_sensor_collector/src/checkPoint") \
.options(table=_table,keyspace=_keySpace) \
.outputMode('append') \
.start()
ds.awaitTermination()
application Desigh Pattern is Master-Slave, 1master-3slave.
why auto off spark-cassandra??
精彩评论