projectId STRING Required

The project which owns the WorkItem's job

jobId STRING Required

The job which the WorkItem is part of

workItemStatuses[] OBJECT

Conveys a worker's progress through the work described by a WorkItem

workItemStatuses[].counterUpdates[] OBJECT

An update to a Counter sent from a worker

workItemStatuses[].counterUpdates[].shortId INTEGER

The service-generated short identifier for this counter. The short_id -> (name, metadata) mapping is constant for the lifetime of a job

workItemStatuses[].counterUpdates[].integerGauge OBJECT

A metric value representing temporal values of a variable

workItemStatuses[].counterUpdates[].floatingPointList OBJECT

A metric value representing a list of floating point numbers

workItemStatuses[].counterUpdates[].integer OBJECT

A representation of an int64, n, that is immune to precision loss when encoded in JSON

workItemStatuses[].counterUpdates[].integerList OBJECT

A metric value representing a list of integers

workItemStatuses[].counterUpdates[].structuredNameAndMetadata OBJECT

A single message which encapsulates structured name and metadata for a given counter

workItemStatuses[].counterUpdates[].floatingPoint NUMBER

Floating point value for Sum, Max, Min

workItemStatuses[].counterUpdates[].integerMean OBJECT

A representation of an integer mean metric contribution

workItemStatuses[].counterUpdates[].internal ANY

Value for internally-defined counters used by the Dataflow service

workItemStatuses[].counterUpdates[].cumulative BOOLEAN

True if this counter is reported as the total cumulative aggregate value accumulated since the worker started working on this WorkItem. By default this is false, indicating that this counter is reported as a delta

workItemStatuses[].counterUpdates[].floatingPointMean OBJECT

A representation of a floating point mean metric contribution

workItemStatuses[].counterUpdates[].boolean BOOLEAN

Boolean value for And, Or

workItemStatuses[].counterUpdates[].nameAndKind OBJECT

Basic metadata about a counter

workItemStatuses[].counterUpdates[].stringList OBJECT

A metric value representing a list of strings

workItemStatuses[].counterUpdates[].distribution OBJECT

A metric value representing a distribution

workItemStatuses[].workItemId STRING

Identifies the WorkItem

workItemStatuses[].errors[] OBJECT

The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details.

You can find out more about this error model and how to work with it in the API Design Guide

workItemStatuses[].errors[].details[] OBJECT

workItemStatuses[].errors[].details[].customKey.value ANY Required

workItemStatuses[].errors[].code INTEGER

The status code, which should be an enum value of google.rpc.Code

workItemStatuses[].errors[].message STRING

A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client

workItemStatuses[].metricUpdates[] OBJECT

Describes the state of a metric

workItemStatuses[].metricUpdates[].scalar ANY

Worker-computed aggregate value for aggregation kinds "Sum", "Max", "Min", "And", and "Or". The possible value types are Long, Double, and Boolean

workItemStatuses[].metricUpdates[].meanCount ANY

Worker-computed aggregate value for the "Mean" aggregation kind. This holds the count of the aggregated values and is used in combination with mean_sum above to obtain the actual mean aggregate value. The only possible value type is Long

workItemStatuses[].metricUpdates[].meanSum ANY

Worker-computed aggregate value for the "Mean" aggregation kind. This holds the sum of the aggregated values and is used in combination with mean_count below to obtain the actual mean aggregate value. The only possible value types are Long and Double

workItemStatuses[].metricUpdates[].updateTime ANY

Timestamp associated with the metric value. Optional when workers are reporting work progress; it will be filled in responses from the metrics API

workItemStatuses[].metricUpdates[].name OBJECT

Identifies a metric, by describing the source which generated the metric

workItemStatuses[].metricUpdates[].distribution ANY

A struct value describing properties of a distribution of numeric values

workItemStatuses[].metricUpdates[].set ANY

Worker-computed aggregate value for the "Set" aggregation kind. The only possible value type is a list of Values whose type can be Long, Double, or String, according to the metric's type. All Values in the list must be of the same type

workItemStatuses[].metricUpdates[].gauge ANY

A struct value describing properties of a Gauge. Metrics of gauge type show the value of a metric across time, and is aggregated based on the newest value

workItemStatuses[].metricUpdates[].internal ANY

Worker-computed aggregate value for internal use by the Dataflow service

workItemStatuses[].metricUpdates[].cumulative BOOLEAN

True if this metric is reported as the total cumulative aggregate value accumulated since the worker started working on this WorkItem. By default this is false, indicating that this metric is reported as a delta that is not associated with any WorkItem

workItemStatuses[].metricUpdates[].kind STRING

Metric aggregation kind. The possible metric aggregation kinds are "Sum", "Max", "Min", "Mean", "Set", "And", "Or", and "Distribution". The specified aggregation kind is case-insensitive.

If omitted, this is not an aggregated value but instead a single metric sample value

workItemStatuses[].dynamicSourceSplit OBJECT

When a task splits using WorkItemStatus.dynamic_source_split, this message describes the two parts of the split relative to the description of the current task's input

workItemStatuses[].dynamicSourceSplit.residual OBJECT

Specification of one of the bundles produced as a result of splitting a Source (e.g. when executing a SourceSplitRequest, or when splitting an active task using WorkItemStatus.dynamic_source_split), relative to the source being split

workItemStatuses[].dynamicSourceSplit.residual.derivationMode ENUMERATION

What source to base the produced source on (if any)

workItemStatuses[].dynamicSourceSplit.primary OBJECT

Specification of one of the bundles produced as a result of splitting a Source (e.g. when executing a SourceSplitRequest, or when splitting an active task using WorkItemStatus.dynamic_source_split), relative to the source being split

workItemStatuses[].dynamicSourceSplit.primary.derivationMode ENUMERATION

What source to base the produced source on (if any)

workItemStatuses[].sourceOperationResponse OBJECT

The result of a SourceOperationRequest, specified in ReportWorkItemStatusRequest.source_operation when the work item is completed

workItemStatuses[].sourceOperationResponse.getMetadata OBJECT

The result of a SourceGetMetadataOperation

workItemStatuses[].sourceOperationResponse.split OBJECT

The response to a SourceSplitRequest

workItemStatuses[].sourceOperationResponse.split.outcome ENUMERATION

Indicates whether splitting happened and produced a list of bundles. If this is USE_CURRENT_SOURCE_AS_IS, the current source should be processed "as is" without splitting. "bundles" is ignored in this case. If this is SPLITTING_HAPPENED, then "bundles" contains a list of bundles into which the source was split

workItemStatuses[].progress OBJECT

Obsolete in favor of ApproximateReportedProgress and ApproximateSplitRequest

workItemStatuses[].progress.percentComplete FLOAT

Obsolete

workItemStatuses[].progress.remainingTime ANY

Obsolete

workItemStatuses[].progress.position OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemStatuses[].progress.position.recordIndex INTEGER

Position is a record index

workItemStatuses[].progress.position.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemStatuses[].progress.position.byteOffset INTEGER

Position is a byte offset

workItemStatuses[].progress.position.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemStatuses[].progress.position.key STRING

Position is a string key, ordered lexicographically

workItemStatuses[].requestedLeaseDuration ANY

Amount of time the worker requests for its lease

workItemStatuses[].reportIndex INTEGER

The report index. When a WorkItem is leased, the lease will contain an initial report index. When a WorkItem's status is reported to the system, the report should be sent with that report index, and the response will contain the index the worker should use for the next report. Reports received with unexpected index values will be rejected by the service.

In order to preserve idempotency, the worker should not alter the contents of a report, even if the worker must submit the same report multiple times before getting back a response. The worker should not submit a subsequent report until the response for the previous report had been received from the service

workItemStatuses[].stopPosition OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemStatuses[].stopPosition.recordIndex INTEGER

Position is a record index

workItemStatuses[].stopPosition.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemStatuses[].stopPosition.concatPosition OBJECT

A position that encapsulates an inner position and an index for the inner position. A ConcatPosition can be used by a reader of a source that encapsulates a set of other sources

workItemStatuses[].stopPosition.concatPosition.index INTEGER

Index of the inner source

workItemStatuses[].stopPosition.byteOffset INTEGER

Position is a byte offset

workItemStatuses[].stopPosition.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemStatuses[].stopPosition.key STRING

Position is a string key, ordered lexicographically

workItemStatuses[].completed BOOLEAN

True if the WorkItem was completed (successfully or unsuccessfully)

workItemStatuses[].reportedProgress OBJECT

A progress measurement of a WorkItem by a worker

workItemStatuses[].reportedProgress.remainingParallelism OBJECT

Represents the level of parallelism in a WorkItem's input, reported by the worker

workItemStatuses[].reportedProgress.remainingParallelism.isInfinite BOOLEAN

Specifies whether the parallelism is infinite. If true, "value" is ignored. Infinite parallelism means the service will assume that the work item can always be split into more non-empty work items by dynamic splitting. This is a work-around for lack of support for infinity by the current JSON-based Java RPC stack

workItemStatuses[].reportedProgress.remainingParallelism.value NUMBER

Specifies the level of parallelism in case it is finite

workItemStatuses[].reportedProgress.position OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemStatuses[].reportedProgress.position.recordIndex INTEGER

Position is a record index

workItemStatuses[].reportedProgress.position.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemStatuses[].reportedProgress.position.byteOffset INTEGER

Position is a byte offset

workItemStatuses[].reportedProgress.position.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemStatuses[].reportedProgress.position.key STRING

Position is a string key, ordered lexicographically

workItemStatuses[].reportedProgress.fractionConsumed NUMBER

Completion as fraction of the input consumed, from 0.0 (beginning, nothing consumed), to 1.0 (end of the input, entire input consumed)

workItemStatuses[].reportedProgress.consumedParallelism OBJECT

Represents the level of parallelism in a WorkItem's input, reported by the worker

workItemStatuses[].reportedProgress.consumedParallelism.isInfinite BOOLEAN

Specifies whether the parallelism is infinite. If true, "value" is ignored. Infinite parallelism means the service will assume that the work item can always be split into more non-empty work items by dynamic splitting. This is a work-around for lack of support for infinity by the current JSON-based Java RPC stack

workItemStatuses[].reportedProgress.consumedParallelism.value NUMBER

Specifies the level of parallelism in case it is finite

workItemStatuses[].sourceFork OBJECT

DEPRECATED in favor of DynamicSourceSplit

workItemStatuses[].sourceFork.residualSource OBJECT

Specification of one of the bundles produced as a result of splitting a Source (e.g. when executing a SourceSplitRequest, or when splitting an active task using WorkItemStatus.dynamic_source_split), relative to the source being split

workItemStatuses[].sourceFork.residualSource.derivationMode ENUMERATION

What source to base the produced source on (if any)

workItemStatuses[].sourceFork.primary OBJECT

DEPRECATED in favor of DerivedSource

workItemStatuses[].sourceFork.primary.derivationMode ENUMERATION

DEPRECATED

workItemStatuses[].sourceFork.primarySource OBJECT

Specification of one of the bundles produced as a result of splitting a Source (e.g. when executing a SourceSplitRequest, or when splitting an active task using WorkItemStatus.dynamic_source_split), relative to the source being split

workItemStatuses[].sourceFork.primarySource.derivationMode ENUMERATION

What source to base the produced source on (if any)

workItemStatuses[].sourceFork.residual OBJECT

DEPRECATED in favor of DerivedSource

workItemStatuses[].sourceFork.residual.derivationMode ENUMERATION

DEPRECATED

workItemStatuses[].totalThrottlerWaitTimeSeconds NUMBER

Total time the worker spent being throttled by external systems

unifiedWorkerRequest OBJECT

Untranslated bag-of-bytes WorkProgressUpdateRequest from UnifiedWorker

unifiedWorkerRequest.customKey.value ANY Required

Untranslated bag-of-bytes WorkProgressUpdateRequest from UnifiedWorker

workerId STRING

The ID of the worker reporting the WorkItem status. If this does not match the ID of the worker which the Dataflow service believes currently has the lease on the WorkItem, the report will be dropped (with an error response)

currentWorkerTime ANY

The current timestamp at the worker

location STRING

The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that contains the WorkItem's job

workItemServiceStates[] OBJECT

The Dataflow service's idea of the current state of a WorkItem being processed by a worker

workItemServiceStates[].suggestedStopPosition OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemServiceStates[].suggestedStopPosition.recordIndex INTEGER

Position is a record index

workItemServiceStates[].suggestedStopPosition.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemServiceStates[].suggestedStopPosition.concatPosition OBJECT

A position that encapsulates an inner position and an index for the inner position. A ConcatPosition can be used by a reader of a source that encapsulates a set of other sources

workItemServiceStates[].suggestedStopPosition.concatPosition.index INTEGER

Index of the inner source

workItemServiceStates[].suggestedStopPosition.byteOffset INTEGER

Position is a byte offset

workItemServiceStates[].suggestedStopPosition.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemServiceStates[].suggestedStopPosition.key STRING

Position is a string key, ordered lexicographically

workItemServiceStates[].reportStatusInterval ANY

New recommended reporting interval

workItemServiceStates[].harnessData OBJECT

Other data returned by the service, specific to the particular worker harness

workItemServiceStates[].harnessData.customKey.value ANY

Other data returned by the service, specific to the particular worker harness

workItemServiceStates[].leaseExpireTime ANY

Time at which the current lease will expire

workItemServiceStates[].metricShortId[] OBJECT

The metric short id is returned to the user alongside an offset into ReportWorkItemStatusRequest

workItemServiceStates[].metricShortId[].shortId INTEGER

The service-generated short identifier for the metric

workItemServiceStates[].metricShortId[].metricIndex INTEGER

The index of the corresponding metric in the ReportWorkItemStatusRequest. Required

workItemServiceStates[].nextReportIndex INTEGER

The index value to use for the next report sent by the worker. Note: If the report call fails for whatever reason, the worker should reuse this index for subsequent report attempts

workItemServiceStates[].suggestedStopPoint OBJECT

Obsolete in favor of ApproximateReportedProgress and ApproximateSplitRequest

workItemServiceStates[].suggestedStopPoint.percentComplete FLOAT

Obsolete

workItemServiceStates[].suggestedStopPoint.remainingTime ANY

Obsolete

workItemServiceStates[].suggestedStopPoint.position OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemServiceStates[].suggestedStopPoint.position.recordIndex INTEGER

Position is a record index

workItemServiceStates[].suggestedStopPoint.position.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemServiceStates[].suggestedStopPoint.position.byteOffset INTEGER

Position is a byte offset

workItemServiceStates[].suggestedStopPoint.position.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemServiceStates[].suggestedStopPoint.position.key STRING

Position is a string key, ordered lexicographically

workItemServiceStates[].hotKeyDetection OBJECT

Proto describing a hot key detected on a given WorkItem

workItemServiceStates[].hotKeyDetection.hotKeyAge ANY

The age of the hot key measured from when it was first detected

workItemServiceStates[].hotKeyDetection.userStepName STRING

User-provided name of the step that contains this hot key

workItemServiceStates[].hotKeyDetection.systemName STRING

System-defined name of the step containing this hot key. Unique across the workflow

workItemServiceStates[].splitRequest OBJECT

A suggestion by the service to the worker to dynamically split the WorkItem

workItemServiceStates[].splitRequest.position OBJECT

Position defines a position within a collection of data. The value can be either the end position, a key (used with ordered collections), a byte offset, or a record index

workItemServiceStates[].splitRequest.position.recordIndex INTEGER

Position is a record index

workItemServiceStates[].splitRequest.position.shufflePosition STRING

CloudPosition is a base64 encoded BatchShufflePosition (with FIXED sharding)

workItemServiceStates[].splitRequest.position.byteOffset INTEGER

Position is a byte offset

workItemServiceStates[].splitRequest.position.end BOOLEAN

Position is past all other positions. Also useful for the end position of an unbounded range

workItemServiceStates[].splitRequest.position.key STRING

Position is a string key, ordered lexicographically

workItemServiceStates[].splitRequest.fractionOfRemainder NUMBER

The fraction of the remainder of work to split the work item at, from 0.0 (split at the current position) to 1.0 (end of the input)

workItemServiceStates[].splitRequest.fractionConsumed NUMBER

A fraction at which to split the work item, from 0.0 (beginning of the input) to 1.0 (end of the input)

unifiedWorkerResponse OBJECT

Untranslated bag-of-bytes WorkProgressUpdateResponse for UnifiedWorker

unifiedWorkerResponse.customKey.value ANY

Untranslated bag-of-bytes WorkProgressUpdateResponse for UnifiedWorker