projectId STRING Required

The project to send the WorkerMessages to

location STRING

The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that contains the job

workerMessages[] OBJECT

WorkerMessage provides information to the backend about a worker

workerMessages[].workerHealthReport OBJECT

WorkerHealthReport contains information about the health of a worker.

The VM should be identified by the labels attached to the WorkerMessage that this health ping belongs to

workerMessages[].workerHealthReport.vmStartupTime ANY

The time the VM was booted

workerMessages[].workerHealthReport.vmIsBroken BOOLEAN

Whether the VM is in a permanently broken state. Broken VMs should be abandoned or deleted ASAP to avoid assigning or completing any work

workerMessages[].workerHealthReport.msg STRING

A message describing any unusual health reports

workerMessages[].workerHealthReport.pods[] OBJECT

workerMessages[].workerHealthReport.pods[].customKey.value ANY Required

workerMessages[].workerHealthReport.vmIsHealthy BOOLEAN

Whether the VM is currently healthy

workerMessages[].workerHealthReport.reportInterval ANY

The interval at which the worker is sending health reports. The default value of 0 should be interpreted as the field is not being explicitly set by the worker

workerMessages[].workerMessageCode OBJECT

A message code is used to report status and error messages to the service. The message codes are intended to be machine readable. The service will take care of translating these into user understandable messages if necessary.

Example use cases:

1. Worker processes reporting successful startup.
2. Worker processes reporting specific errors (e.g. package staging failure).

workerMessages[].workerMessageCode.parameters OBJECT

Parameters contains specific information about the code.

This is a struct to allow parameters of different types.

Examples:

1. For a "HARNESS_STARTED" message parameters might provide the name of the worker and additional data like timing information.
2. For a "GCS_DOWNLOAD_ERROR" parameters might contain fields listing the GCS objects being downloaded and fields containing errors.

In general complex data structures should be avoided. If a worker needs to send a specific and complicated data structure then please consider defining a new proto and adding it to the data oneof in WorkerMessageResponse.

Conventions: Parameters should only be used for information that isn't typically passed as a label. hostname and other worker identifiers should almost always be passed as labels since they will be included on most messages

workerMessages[].workerMessageCode.parameters.customKey.value ANY Required

Parameters contains specific information about the code.

This is a struct to allow parameters of different types.

Examples:

1. For a "HARNESS_STARTED" message parameters might provide the name of the worker and additional data like timing information.
2. For a "GCS_DOWNLOAD_ERROR" parameters might contain fields listing the GCS objects being downloaded and fields containing errors.

In general complex data structures should be avoided. If a worker needs to send a specific and complicated data structure then please consider defining a new proto and adding it to the data oneof in WorkerMessageResponse.

Conventions: Parameters should only be used for information that isn't typically passed as a label. hostname and other worker identifiers should almost always be passed as labels since they will be included on most messages

workerMessages[].workerMessageCode.code STRING

The code is a string intended for consumption by a machine that identifies the type of message being sent. Examples:

1. "HARNESS_STARTED" might be used to indicate the worker harness has started.
2. "GCS_DOWNLOAD_ERROR" might be used to indicate an error downloading a GCS file as part of the boot process of one of the worker containers.

This is a string and not an enum to make it easy to add new codes without waiting for an API change

workerMessages[].workerMetrics OBJECT

Worker metrics exported from workers. This contains resource utilization metrics accumulated from a variety of sources. For more information, see go/df-resource-signals

workerMessages[].workerMetrics.cpuTime[] OBJECT

Modeled after information exposed by /proc/stat

workerMessages[].labels OBJECT

Labels are used to group WorkerMessages. For example, a worker_message about a particular container might have the labels: { "JOB_ID": "2015-04-22", "WORKER_ID": "wordcount-vm-2015…" "CONTAINER_TYPE": "worker", "CONTAINER_ID": "ac1234def"} Label tags typically correspond to Label enum values. However, for ease of development other strings can be used as tags. LABEL_UNSPECIFIED should not be used here

workerMessages[].labels.customKey.value STRING Required

Labels are used to group WorkerMessages. For example, a worker_message about a particular container might have the labels: { "JOB_ID": "2015-04-22", "WORKER_ID": "wordcount-vm-2015…" "CONTAINER_TYPE": "worker", "CONTAINER_ID": "ac1234def"} Label tags typically correspond to Label enum values. However, for ease of development other strings can be used as tags. LABEL_UNSPECIFIED should not be used here

workerMessages[].time ANY

The timestamp of the worker_message

workerMessages[].workerLifecycleEvent OBJECT

A report of an event in a worker's lifecycle. The proto contains one event, because the worker is expected to asynchronously send each message immediately after the event. Due to this asynchrony, messages may arrive out of order (or missing), and it is up to the consumer to interpret. The timestamp of the event is in the enclosing WorkerMessage proto

workerMessages[].workerLifecycleEvent.event ENUMERATION

The event being reported

workerMessages[].workerLifecycleEvent.containerStartTime ANY

The start time of this container. All events will report this so that events can be grouped together across container/VM restarts

workerMessages[].workerLifecycleEvent.metadata OBJECT

Other stats that can accompany an event. E.g. { "downloaded_bytes" : "123456" }

workerMessages[].workerLifecycleEvent.metadata.customKey.value STRING Required

Other stats that can accompany an event. E.g. { "downloaded_bytes" : "123456" }

workerMessages[].workerShutdownNotice OBJECT

Shutdown notification from workers. This is to be sent by the shutdown script of the worker VM so that the backend knows that the VM is being shut down

workerMessages[].workerShutdownNotice.reason STRING

The reason for the worker shutdown. Current possible values are: "UNKNOWN": shutdown reason is unknown. "PREEMPTION": shutdown reason is preemption. Other possible reasons may be added in the future

workerMessageResponses[] OBJECT

A worker_message response allows the server to pass information to the sender

workerMessageResponses[].workerMetricsResponse OBJECT

Service-side response to WorkerMessage reporting resource utilization

workerMessageResponses[].workerShutdownNoticeResponse OBJECT

Service-side response to WorkerMessage issuing shutdown notice

workerMessageResponses[].workerHealthReportResponse OBJECT

WorkerHealthReportResponse contains information returned to the worker in response to a health ping

workerMessageResponses[].workerHealthReportResponse.reportInterval ANY

A positive value indicates the worker should change its reporting interval to the specified value.

The default value of zero means no change in report rate is requested by the server