org.apache.beam.sdk.io.kafka

Class KafkaIO.Write<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

org.apache.beam.sdk.io.kafka.KafkaIO.Write<K,V>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

KafkaIO

public abstract static class KafkaIO.Write<K,V>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

A PTransform to write to a Kafka topic with KVs . See KafkaIO for more information on usage and configuration.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaIO.Write.External Exposes KafkaIO.Write as an external transform for cross-language usage.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PDone	expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>> input)
void	populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
KafkaIO.Write<K,V>	updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.
void	validate(@Nullable org.apache.beam.sdk.options.PipelineOptions options)
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<V>,org.apache.beam.sdk.values.PDone>	values() Writes just the values to Kafka.
KafkaIO.Write<K,V>	withBootstrapServers(java.lang.String bootstrapServers) Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withConsumerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends org.apache.kafka.clients.consumer.Consumer<?,?>> consumerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withEOS(int numShards, java.lang.String sinkGroupId) Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withInputTimestamp() Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withKeySerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<K>> keySerializer) Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Update configuration for the producer.
KafkaIO.Write<K,V>	withProducerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,org.apache.kafka.clients.producer.Producer<K,V>> producerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withPublishTimestampFunction(KafkaPublishTimestampFunction<org.apache.beam.sdk.values.KV<K,V>> timestampFunction) Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.
KafkaIO.Write<K,V>	withTopic(java.lang.String topic) Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withValueSerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<V>> valueSerializer) Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withBootstrapServers

public KafkaIO.Write<K,V> withBootstrapServers(java.lang.String bootstrapServers)

Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.

withTopic

public KafkaIO.Write<K,V> withTopic(java.lang.String topic)

Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.

withKeySerializer

public KafkaIO.Write<K,V> withKeySerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<K>> keySerializer)

Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.

withValueSerializer

public KafkaIO.Write<K,V> withValueSerializer(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<V>> valueSerializer)

Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

withProducerFactoryFn

public KafkaIO.Write<K,V> withProducerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,org.apache.kafka.clients.producer.Producer<K,V>> producerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

withInputTimestamp

public KafkaIO.Write<K,V> withInputTimestamp()

Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.

withPublishTimestampFunction

@Deprecated
public KafkaIO.Write<K,V> withPublishTimestampFunction(KafkaPublishTimestampFunction<org.apache.beam.sdk.values.KV<K,V>> timestampFunction)

Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.

Wrapper method over KafkaIO.WriteRecords.withPublishTimestampFunction(KafkaPublishTimestampFunction), used to keep the compatibility with old API based on KV type of element.

withEOS

public KafkaIO.Write<K,V> withEOS(int numShards,
                                  java.lang.String sinkGroupId)

Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.

withConsumerFactoryFn

public KafkaIO.Write<K,V> withConsumerFactoryFn(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends org.apache.kafka.clients.consumer.Consumer<?,?>> consumerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

updateProducerProperties

@Deprecated
public KafkaIO.Write<K,V> updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.

Adds the given producer properties, overriding old values of properties with the same key.

withProducerConfigUpdates

public KafkaIO.Write<K,V> withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Update configuration for the producer. Note that the default producer properties will not be completely overridden. This method only updates the value which has the same key.

By default, the producer uses the configuration from KafkaIO.WriteRecords.DEFAULT_PRODUCER_PROPERTIES.

expand

public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

validate

public void validate(@Nullable org.apache.beam.sdk.options.PipelineOptions options)

Overrides:

validate in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

populateDisplayData

public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)

Specified by:

populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData

Overrides:

populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,V>>,org.apache.beam.sdk.values.PDone>

values

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<V>,org.apache.beam.sdk.values.PDone> values()

Writes just the values to Kafka. This is useful for writing collections of values rather thank KVs.