org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies

Class DefaultRollingPolicy<IN,BucketID>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.DefaultRollingPolicy<IN,BucketID>

All Implemented Interfaces:

Serializable, RollingPolicy<IN,BucketID>

@PublicEvolving
public final class DefaultRollingPolicy<IN,BucketID>
extends Object
implements RollingPolicy<IN,BucketID>

The default implementation of the RollingPolicy.

This policy rolls a part file if:

1. there is no open part file,
2. the current file has reached the maximum bucket size (by default 128MB),
3. the current file is older than the roll over interval (by default 60 sec), or
4. the current file has not been written to for more than the allowed inactivityTime (by default 60 sec).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultRollingPolicy.PolicyBuilder A helper class that holds the configuration properties for the DefaultRollingPolicy.

Method Summary

Modifier and Type	Method and Description
static DefaultRollingPolicy.PolicyBuilder	builder() Creates a new DefaultRollingPolicy.PolicyBuilder that is used to configure and build an instance of DefaultRollingPolicy.
static DefaultRollingPolicy.PolicyBuilder	create() Deprecated.
long	getInactivityInterval() Returns time duration of allowed inactivity after which a part file will have to roll.
long	getMaxPartSize() Returns the maximum part file size before rolling.
long	getRolloverInterval() Returns the maximum time duration a part file can stay open before rolling.
boolean	shouldRollOnCheckpoint(PartFileInfo<BucketID> partFileState) Determines if the in-progress part file for a bucket should roll on every checkpoint.
boolean	shouldRollOnEvent(PartFileInfo<BucketID> partFileState, IN element) Determines if the in-progress part file for a bucket should roll based on its current state, e.g.
boolean	shouldRollOnProcessingTime(PartFileInfo<BucketID> partFileState, long currentTime) Determines if the in-progress part file for a bucket should roll based on a time condition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

shouldRollOnCheckpoint

public boolean shouldRollOnCheckpoint(PartFileInfo<BucketID> partFileState)
                               throws IOException

Description copied from interface: RollingPolicy

Determines if the in-progress part file for a bucket should roll on every checkpoint.

Specified by:

shouldRollOnCheckpoint in interface RollingPolicy<IN,BucketID>

Parameters:

partFileState - the state of the currently open part file of the bucket.

Returns:

True if the part file should roll, false otherwise.

Throws:

IOException

shouldRollOnEvent

public boolean shouldRollOnEvent(PartFileInfo<BucketID> partFileState,
                                 IN element)
                          throws IOException

Description copied from interface: RollingPolicy

Determines if the in-progress part file for a bucket should roll based on its current state, e.g. its size.

Specified by:

shouldRollOnEvent in interface RollingPolicy<IN,BucketID>

Parameters:

partFileState - the state of the currently open part file of the bucket.

element - the element being processed.

Returns:

True if the part file should roll, false otherwise.

Throws:

IOException

shouldRollOnProcessingTime

public boolean shouldRollOnProcessingTime(PartFileInfo<BucketID> partFileState,
                                          long currentTime)

Description copied from interface: RollingPolicy

Determines if the in-progress part file for a bucket should roll based on a time condition.

Specified by:

shouldRollOnProcessingTime in interface RollingPolicy<IN,BucketID>

Parameters:

partFileState - the state of the currently open part file of the bucket.

currentTime - the current processing time.

Returns:

True if the part file should roll, false otherwise.

getMaxPartSize

public long getMaxPartSize()

Returns the maximum part file size before rolling.

Returns:

Max size in bytes

getRolloverInterval

public long getRolloverInterval()

Returns the maximum time duration a part file can stay open before rolling.

Returns:

Time duration in milliseconds

getInactivityInterval

public long getInactivityInterval()

Returns time duration of allowed inactivity after which a part file will have to roll.

Returns:

Time duration in milliseconds

builder

public static DefaultRollingPolicy.PolicyBuilder builder()

Creates a new DefaultRollingPolicy.PolicyBuilder that is used to configure and build an instance of DefaultRollingPolicy.

create

@Deprecated
public static DefaultRollingPolicy.PolicyBuilder create()

Deprecated.

This method is Deprecated, use builder() instead.