Could we add SchemaUpdateOption param to BQ WriteParam?

[arthurenc]

CONTEXT

We are trying to use saveAsBigQueryTable to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do so as the schema for the data we want to append to the table has additional fields not present in the table schema.

"errors" : [ {
      "message" : "Provided Schema does not match Table table_name. Cannot add fields (field: field_name)",
      "reason" : "invalid"
    }

SOLUTION

Google offers a parameter SchemaUpdateOptions which allows you to update the schema of a destination table (docs). This parameter can be configured in two ways; ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION. If we configure it to ALLOW_FIELD_ADDITION then additional nullable fields, that are not already present in the table schema, could be appended using the saveAsBigQueryTable method.

At the moment this is not offered as a configuration option with the scio wrapper. Would it be possible to get this added in?

[RustedBones]

Hi @arthurenc,
Thanks for the API change request.

In the meantime, you can leverage the configOverride to modify the underlying beam IO:

.saveAsBigQueryTable(
  table,
  schema,
  configOverride = _.withSchemaUpdateOptions(myUpdate)
)