Cloud Spanner import/export

Tue, Jan 10, 2023 2-minute read

Spanner import/export

Export

Using dataflow, you can create a job, name it and use a template :
Cloud Spanner to Text Files on Cloud Storage
this will as for :

project id
instance id
database
table
cloud storage path for the files generated
cloud storage temporary path for the process files This creates de csv file and a manifest, the manifest appears to be only showing table columns info:

{"pid":"INT64","item":"STRING(30)","location":"INT64","quantity":"INT64"}

, but in the lab there is a more detailes example:

{
    "tables": [
        {
            "table_name": "current_inventory",
            "file_patterns": [
                "gs://spls/ccl033/inventory.csv"
            ],
            "columns": [
                {"column_name" : "pid", "type_name" : "INT64" },
                {"column_name" : "item", "type_name" : "STRING" },
                {"column_name" : "location", "type_name" : "INT64" },
                {"column_name" : "quantity", "type_name" : "INT64" }
            ]
        }
    ]
}

Import

To import a csv to spanner , in Dataflow: Create a job with template Text files on Cloud Storage to Cloud Spanner
set the project id , instance , database …

If the job does fail, you have to reenable dataflow api:

gcloud services disable dataflow.googleapis.com --force
gcloud services enable dataflow.googleapis.com

And then clone the previously failed job, and run it again.

Tip

Create a bucket with project id name and copy a dummy file to it:

gsutil mb gs://`gcloud config list --format 'value(core.project)' 2>/dev/null`
touch dummy
gsutil cp dummy gs://`gcloud config list --format 'value(core.project)' 2>/dev/null`/tmp/dummy

pktbl