Stance-Detection-in-Web-and-Social-Media/BERT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of BERT FineTuning with Cloud TPU: Sentence and Sentence-Pair Classification Tasks",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkTLZ3I4_7c_",
        "colab_type": "text"
      },
      "source": [
        "# BERT finetuning tasks in 5 minutes with Cloud TPU\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\" >\n",
        " <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wtjs1QDb3DX",
        "colab_type": "text"
      },
      "source": [
        "**BERT**, or **B**idirectional **E**mbedding **R**epresentations from **T**ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.\n",
        "\n",
        "This Colab demonstates using a free Colab Cloud TPU to fine-tune sentence and sentence-pair classification tasks built on top of pretrained BERT models.\n",
        "\n",
        "**Note:**  You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud \n",
        "Storage) bucket for this Colab to run.\n",
        "\n",
        "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) for how to create GCP account and GCS bucket. You have [$300 free credit](https://cloud.google.com/free/) to get started with any GCP product. You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs.\n",
        "\n",
        "Once you finish the setup, let's start!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ycHMh-bhC-vX",
        "colab_type": "text"
      },
      "source": [
        "**Firstly**, we need to set up Colab TPU running environment, verify a TPU device is succesfully connected and upload credentials to TPU for GCS bucket usage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "191zq3ZErihP",
        "colab_type": "code",
        "outputId": "1a6a8e2d-a095-4732-eed5-917acf31b1c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        }
      },
      "source": [
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TPU address is grpc://10.77.133.178:8470\n",
            "TPU devices:\n",
            "[_DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:CPU:0, CPU, -1, 11193819257324014272),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 11743471906924308223),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 8817390715541045037),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 13647766437076739427),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 12291117123528086541),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 5167520003982709723),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 1806457377803209548),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 10367945529533593525),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 11821730786090633668),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 17853583825165986593),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 11451074782555305885)]\n",
            "\n",
            "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUBP35oCDmbF",
        "colab_type": "text"
      },
      "source": [
        "**Secondly**, prepare and import BERT modules."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tEXK_HWODp60",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!rm -rf bert"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wzwke0sxS6W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "\n",
        "!test -d bert || git clone https://github.com/shalmolighosh/bert/\n",
        "if not 'bert' in sys.path:\n",
        "  sys.path += ['bert']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5tDCk6tB-ee",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!cat bert/run_classifier.py | grep -C 10 _read_csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JXoAweIA8kYN",
        "colab_type": "code",
        "outputId": "3d6f6154-4da2-4e8e-9e2d-1f4edcec51e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O3PXnf6E9b5c",
        "colab_type": "code",
        "outputId": "87fe0f1a-516b-4669-cb80-251bd937d5db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!ls gdrive/My\\ Drive/BERT/Data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Atheism  CC  FM  HC  LA\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRu1aKO1D7-Z",
        "colab_type": "text"
      },
      "source": [
        "**Thirdly**, prepare for training:\n",
        "\n",
        "*  Specify task and download training data.\n",
        "*  Specify BERT pretrained model\n",
        "*  Specify GS bucket, create output directory for model checkpoints and eval results.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tYkaAlJNfhul",
        "colab_type": "code",
        "outputId": "e1a98277-8267-4fea-cc83-d528bbcace3c",
        "cellView": "both",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "TASK = 'HC' #@param {type:\"string\"}\n",
        "assert TASK in ('MRPC', 'CoLA','Atheism','CC','HC','LA','FM','ALL'), 'Only (MRPC, CoLA, Sem) are demonstrated here.'\n",
        "# Download glue data.\n",
        "if TASK=='MRPC' or TASK == 'CoLA':\n",
        "  ! test -d download_glue_repo || git clone https://gist.github.com/60c2bdb54d156a41194446737ce03e2e.git download_glue_repo\n",
        "  !python download_glue_repo/download_glue_data.py --data_dir='glue_data' --tasks=$TASK\n",
        "  TASK_DATA_DIR = 'glue_data/' + TASK\n",
        "\n",
        "elif TASK!='ALL':\n",
        "  TASK_DATA_DIR = 'gdrive/My\\ Drive/BERT/Data/' + TASK\n",
        "\n",
        "else:\n",
        "  TASK_DATA_DIR = 'gdrive/My\\ Drive/BERT/Data/'\n",
        "  \n",
        "  \n",
        "print('***** Task data directory: {} *****'.format(TASK_DATA_DIR))\n",
        "!ls $TASK_DATA_DIR\n",
        "\n",
        "# Available pretrained model checkpoints:\n",
        "#   uncased_L-12_H-768_A-12: uncased BERT base model\n",
        "#   uncased_L-24_H-1024_A-16: uncased BERT large model\n",
        "#   cased_L-12_H-768_A-12: cased BERT large model\n",
        "BERT_MODEL = 'uncased_L-24_H-1024_A-16' #@param {type:\"string\"}\n",
        "BERT_PRETRAINED_DIR = 'gs://cloud-tpu-checkpoints/bert/' + BERT_MODEL\n",
        "print('***** BERT pretrained directory: {} *****'.format(BERT_PRETRAINED_DIR))\n",
        "!gsutil ls $BERT_PRETRAINED_DIR\n",
        "\n",
        "BUCKET = 'bert-large-pair' #@param {type:\"string\"}\n",
        "assert BUCKET, 'Must specify an existing GCS bucket name'\n",
        "OUTPUT_DIR = 'gs://{}/bert/models/{}/{}_new'.format(BUCKET,BERT_MODEL ,TASK)\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***** Task data directory: gdrive/My\\ Drive/BERT/Data/HC *****\n",
            "test_preprocessed.csv  train_preprocessed.csv\n",
            "***** BERT pretrained directory: gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16 *****\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_config.json\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.data-00000-of-00001\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.index\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.meta\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/checkpoint\n",
            "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/vocab.txt\n",
            "***** Model output directory: gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qpn1MM6dd7YS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!gsutil cp gs://bert-final/bert/models/Atheism/* gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/Atheism"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hcpfl4N2EdOk",
        "colab_type": "text"
      },
      "source": [
        "**Now, let's play!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uu2dQ_TId-uH",
        "colab_type": "code",
        "outputId": "0a739553-e406-4059-f3ca-406c3a92d031",
        "cellView": "both",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        }
      },
      "source": [
        "# Setup task specific model and TPU running config.\n",
        "\n",
        "import modeling\n",
        "import optimization\n",
        "import run_classifier\n",
        "import tokenization\n",
        "\n",
        "if TASK!='ALL':\n",
        "  TASK_DATA_DIR = 'gdrive/My Drive/BERT/Data/' + TASK\n",
        "else:\n",
        "  TASK_DATA_DIR = 'gdrive/My Drive/BERT/Data/'\n",
        "  \n",
        "# Model Hyper Parameters\n",
        "TRAIN_BATCH_SIZE = 32\n",
        "EVAL_BATCH_SIZE = 8\n",
        "LEARNING_RATE = 2e-5\n",
        "NUM_TRAIN_EPOCHS = \"11\" #@param {type:\"string\"}\n",
        "NUM_TRAIN_EPOCHS = int(NUM_TRAIN_EPOCHS)\n",
        "WARMUP_PROPORTION = 0.1\n",
        "MAX_SEQ_LENGTH = 128\n",
        "# Model configs\n",
        "SAVE_CHECKPOINTS_STEPS = 1000\n",
        "ITERATIONS_PER_LOOP = 1000\n",
        "NUM_TPU_CORES = 8\n",
        "VOCAB_FILE = os.path.join(BERT_PRETRAINED_DIR, 'vocab.txt')\n",
        "CONFIG_FILE = os.path.join(BERT_PRETRAINED_DIR, 'bert_config.json')\n",
        "INIT_CHECKPOINT = os.path.join(BERT_PRETRAINED_DIR, 'bert_model.ckpt')\n",
        "DO_LOWER_CASE = BERT_MODEL.startswith('uncased')\n",
        "\n",
        "processors = {\n",
        "  \"cola\": run_classifier.ColaProcessor,\n",
        "  \"mnli\": run_classifier.MnliProcessor,\n",
        "  \"mrpc\": run_classifier.MrpcProcessor,\n",
        "  \"hc\"  : run_classifier.SemProcessor,\n",
        "    \"atheism\" : run_classifier.SemProcessor,\n",
        "    \"fm\" : run_classifier.SemProcessor,\n",
        "    \"cc\" : run_classifier.SemProcessor,\n",
        "    \"la\" : run_classifier.SemProcessor,\n",
        "    \"all\" : run_classifier.SemProcessor\n",
        "}\n",
        "print(processors[TASK.lower()])\n",
        "\n",
        "tokenizer = tokenization.FullTokenizer(vocab_file=VOCAB_FILE, do_lower_case=DO_LOWER_CASE)\n",
        "\n",
        "tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "run_config = tf.contrib.tpu.RunConfig(\n",
        "    cluster=tpu_cluster_resolver,\n",
        "    model_dir=OUTPUT_DIR,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,\n",
        "    tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "        iterations_per_loop=ITERATIONS_PER_LOOP,\n",
        "        num_shards=NUM_TPU_CORES,\n",
        "        per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))\n",
        "\n",
        "if TASK == 'ALL':\n",
        "  train_examples = []\n",
        "  full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n",
        "  for key,value in full_forms.items():\n",
        "    processor = run_classifier.SemProcessor(use_pair=True, topic = value)\n",
        "    label_list = processor.get_labels()\n",
        "    train_examples += processor.get_train_examples(TASK_DATA_DIR+key)   \n",
        "\n",
        "else:\n",
        "  full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n",
        "  processor = processors[TASK.lower()](use_pair=False,\\\n",
        "                                       topic=full_forms[TASK])\n",
        "  label_list = processor.get_labels()\n",
        "  train_examples = processor.get_train_examples(TASK_DATA_DIR)\n",
        "\n",
        "\n",
        "print(\"Number of train examples :\",len(train_examples))\n",
        "  \n",
        "num_train_steps = int(\n",
        "    len(train_examples) / TRAIN_BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
        "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)\n",
        "\n",
        "model_fn = run_classifier.model_fn_builder(\n",
        "    bert_config=modeling.BertConfig.from_json_file(CONFIG_FILE),\n",
        "    num_labels=len(label_list),\n",
        "    init_checkpoint=INIT_CHECKPOINT,\n",
        "    learning_rate=LEARNING_RATE,\n",
        "    num_train_steps=num_train_steps,\n",
        "    num_warmup_steps=num_warmup_steps,\n",
        "    use_tpu=True,\n",
        "    use_one_hot_embeddings=True)\n",
        "\n",
        "estimator = tf.contrib.tpu.TPUEstimator(\n",
        "    use_tpu=True,\n",
        "    model_fn=model_fn,\n",
        "    config=run_config,\n",
        "    train_batch_size=TRAIN_BATCH_SIZE,\n",
        "    eval_batch_size=EVAL_BATCH_SIZE)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'run_classifier.SemProcessor'>\n",
            "Number of train examples : 639\n",
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcce37cf9d8>) includes params argument, but params are not passed to Estimator.\n",
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.77.133.178:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcce37e1208>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.77.133.178:8470', '_evaluation_master': 'grpc://10.77.133.178:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcce3b79320>}\n",
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HCHohyPJFtmc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import csv\n",
        "def read_csv(input_file, quotechar=None):\n",
        "  \"\"\"Reads a tab separated value file.\"\"\"\n",
        "  with tf.gfile.Open(input_file, \"r\") as f:\n",
        "    reader = reader = csv.reader(f)\n",
        "    lines = []\n",
        "    for line in reader:\n",
        "      if sys.version_info[0]==2:\n",
        "        line = list(unicode(cell, 'utf-8') for cell in line)\n",
        "      lines.append(line)\n",
        "    return lines\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kqPahgY9GMQK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#lines = read_csv(TASK_DATA_DIR+'/train_preprocessed.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TTvlyZ7KGhMM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#lines"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5U_c8s2AvhgL",
        "colab_type": "code",
        "outputId": "450c3bb7-e7c4-47d5-9eaf-a5a66dc61aa6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 766
        }
      },
      "source": [
        "# Train the model.\n",
        "print('MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...')\n",
        "train_features = run_classifier.convert_examples_to_features(\n",
        "    train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "print('***** Started training at {} *****'.format(datetime.datetime.now()))\n",
        "print('  Num examples = {}'.format(len(train_examples)))\n",
        "print('  Batch size = {}'.format(TRAIN_BATCH_SIZE))\n",
        "tf.logging.info(\"  Num steps = %d\", num_train_steps)\n",
        "train_input_fn = run_classifier.input_fn_builder(\n",
        "    features=train_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=True,\n",
        "    drop_remainder=True)\n",
        "#estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "print('***** Finished training at {} *****'.format(datetime.datetime.now()))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...\n",
            "INFO:tensorflow:Writing example 0 of 639\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: train-0\n",
            "INFO:tensorflow:tokens: [CLS] rt gunn jessica because i want young american women to be able to be proud of the 1st woman president [SEP]\n",
            "INFO:tensorflow:input_ids: 101 19387 22079 8201 2138 1045 2215 2402 2137 2308 2000 2022 2583 2000 2022 7098 1997 1996 3083 2450 2343 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 1 (id = 1)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: train-1\n",
            "INFO:tensorflow:tokens: [CLS] chris 1791 news buster ##s too years ago # hillary # never answered whether she used private email # liberal # media passed on reporting [SEP]\n",
            "INFO:tensorflow:input_ids: 101 3782 14362 2739 18396 2015 2205 2086 3283 1001 18520 1001 2196 4660 3251 2016 2109 2797 10373 1001 4314 1001 2865 2979 2006 7316 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: train-2\n",
            "INFO:tensorflow:tokens: [CLS] d new ha ##user take that , marco rub ##io go ##p cannot pan ##der to the latino vote while pushing policies that hurt the community most [SEP]\n",
            "INFO:tensorflow:input_ids: 101 1040 2047 5292 20330 2202 2008 1010 8879 14548 3695 2175 2361 3685 6090 4063 2000 1996 7402 3789 2096 6183 6043 2008 3480 1996 2451 2087 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 2 (id = 2)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: train-3\n",
            "INFO:tensorflow:tokens: [CLS] state dept am ##b cathy russell adam smith usa hillary clinton hillary for i a barack obama the whole # world # is supporting you [SEP]\n",
            "INFO:tensorflow:input_ids: 101 2110 29466 2572 2497 18305 5735 4205 3044 3915 18520 7207 18520 2005 1045 1037 13857 8112 1996 2878 1001 2088 1001 2003 4637 2017 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 1 (id = 1)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: train-4\n",
            "INFO:tensorflow:tokens: [CLS] lets remember # dick cheney # is an un ##ind ##ic ##ted war criminal before we start yelling # ben ##gh ##azi # day after day will we ever see justice ? [SEP]\n",
            "INFO:tensorflow:input_ids: 101 11082 3342 1001 5980 23745 1001 2003 2019 4895 22254 2594 3064 2162 4735 2077 2057 2707 13175 1001 3841 5603 16103 1001 2154 2044 2154 2097 2057 2412 2156 3425 1029 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 2 (id = 2)\n",
            "***** Started training at 2019-04-13 17:27:42.745922 *****\n",
            "  Num examples = 639\n",
            "  Batch size = 32\n",
            "INFO:tensorflow:  Num steps = 199\n",
            "***** Finished training at 2019-04-13 17:27:42.748669 *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vA-Vf52AGI2r",
        "colab_type": "code",
        "outputId": "89a8ab47-3156-434b-af2a-3052a5175a7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        }
      },
      "source": [
        "estimator = tf.contrib.tpu.TPUEstimator(\n",
        "    use_tpu=True,\n",
        "    model_fn=model_fn,\n",
        "    config=run_config,\n",
        "    train_batch_size=TRAIN_BATCH_SIZE,\n",
        "    eval_batch_size=EVAL_BATCH_SIZE,\n",
        "    predict_batch_size = EVAL_BATCH_SIZE)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcce37cf9d8>) includes params argument, but params are not passed to Estimator.\n",
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.77.133.178:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcce2de4c50>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.77.133.178:8470', '_evaluation_master': 'grpc://10.77.133.178:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcce3b79320>}\n",
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eoXRtSPZvdiS",
        "colab_type": "code",
        "outputId": "bb64befc-a858-4d12-b9e2-497b8a9ac132",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 732
        }
      },
      "source": [
        "# Eval the model.\n",
        "if TASK != 'ALL':\n",
        "  eval_examples = processor.get_dev_examples(TASK_DATA_DIR)\n",
        "  \n",
        "else:\n",
        "  eval_examples = []\n",
        "  full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n",
        "  for key,value in full_forms.items():\n",
        "    processor = run_classifier.SemProcessor(use_pair=True, topic = value)\n",
        "    eval_examples += processor.get_dev_examples(TASK_DATA_DIR+key)   \n",
        "\n",
        "eval_features = run_classifier.convert_examples_to_features(\n",
        "    eval_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "print('***** Started evaluation at {} *****'.format(datetime.datetime.now()))\n",
        "print('  Num examples = {}'.format(len(eval_examples)))\n",
        "print('  Batch size = {}'.format(EVAL_BATCH_SIZE))\n",
        "# Eval will be slightly WRONG on the TPU because it will truncate\n",
        "# the last batch.\n",
        "eval_steps = int(len(eval_examples) / EVAL_BATCH_SIZE)\n",
        "eval_input_fn = run_classifier.input_fn_builder(\n",
        "    features=eval_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=False,\n",
        "    drop_remainder=True)\n",
        "#result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)\n",
        "print('***** Finished evaluation at {} *****'.format(datetime.datetime.now()))\n",
        "#output_eval_file = os.path.join(OUTPUT_DIR, \"eval_results.txt\")\n",
        "#with tf.gfile.GFile(output_eval_file, \"w\") as writer:\n",
        "#  print(\"***** Eval results *****\")\n",
        "#  for key in sorted(result.keys()):\n",
        "#    print('  {} = {}'.format(key, str(result[key])))\n",
        "#    writer.write(\"%s = %s\\n\" % (key, str(result[key])))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 295\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: dev-0\n",
            "INFO:tensorflow:tokens: [CLS] # mt ##p # meet the press how is del ##eti ##ng emails part of the government record different from eras ##ing parts of a tape ? # nixon # # p ##2 # [SEP]\n",
            "INFO:tensorflow:input_ids: 101 1001 11047 2361 1001 3113 1996 2811 2129 2003 3972 20624 3070 22028 2112 1997 1996 2231 2501 2367 2013 28500 2075 3033 1997 1037 6823 1029 1001 11296 1001 1001 1052 2475 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: dev-1\n",
            "INFO:tensorflow:tokens: [CLS] jd son 78 andrew b roe ring andrew why do you care about what i think ? i did not realize that i was this important sir t ##wee ##t andrew is a paid troll [SEP]\n",
            "INFO:tensorflow:input_ids: 101 26219 2365 6275 4080 1038 20944 3614 4080 2339 2079 2017 2729 2055 2054 1045 2228 1029 1045 2106 2025 5382 2008 1045 2001 2023 2590 2909 1056 28394 2102 4080 2003 1037 3825 18792 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: dev-2\n",
            "INFO:tensorflow:tokens: [CLS] the white male vote is solid ##ly go ##p the black vote is solid ##ly them that leaves white females and brown people # feel the bern # [SEP]\n",
            "INFO:tensorflow:input_ids: 101 1996 2317 3287 3789 2003 5024 2135 2175 2361 1996 2304 3789 2003 5024 2135 2068 2008 3727 2317 3801 1998 2829 2111 1001 2514 1996 16595 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: dev-3\n",
            "INFO:tensorflow:tokens: [CLS] ny investing big banker buds need to rat ##chet up their hillary cares about the little people propaganda [SEP]\n",
            "INFO:tensorflow:input_ids: 101 6396 19920 2502 13448 26734 2342 2000 9350 20318 2039 2037 18520 14977 2055 1996 2210 2111 10398 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: dev-4\n",
            "INFO:tensorflow:tokens: [CLS] go ##p why should i believe you on this ? the go ##p leaders in congress wo n ' t fight obama now ! ! ! # t cot # [SEP]\n",
            "INFO:tensorflow:input_ids: 101 2175 2361 2339 2323 1045 2903 2017 2006 2023 1029 1996 2175 2361 4177 1999 3519 24185 1050 1005 1056 2954 8112 2085 999 999 999 1001 1056 26046 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "***** Started evaluation at 2019-04-13 17:28:33.466208 *****\n",
            "  Num examples = 295\n",
            "  Batch size = 8\n",
            "***** Finished evaluation at 2019-04-13 17:28:33.467656 *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gW32URRgenOv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#?? estimator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bjVueAUieoO_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "preds = estimator.predict(\n",
        "              input_fn=eval_input_fn)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ULIjFr-YgAP_",
        "colab_type": "code",
        "outputId": "1f6f495a-6521-459e-bb26-ec8510c3a039",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 9102
        }
      },
      "source": [
        "all_preds = []\n",
        "for pred in preds:\n",
        "  all_preds.append(pred)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Querying Tensorflow master (grpc://10.77.133.178:8470) for TPU system metadata.\n",
            "INFO:tensorflow:Found TPU system:\n",
            "INFO:tensorflow:*** Num TPU Cores: 8\n",
            "INFO:tensorflow:*** Num TPU Workers: 1\n",
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 11193819257324014272)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 11743471906924308223)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 8817390715541045037)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 13647766437076739427)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 12291117123528086541)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 5167520003982709723)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 1806457377803209548)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 10367945529533593525)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 11821730786090633668)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 17853583825165986593)\n",
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 11451074782555305885)\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:*** Features ***\n",
            "INFO:tensorflow:  name = input_ids, shape = (1, 128)\n",
            "INFO:tensorflow:  name = input_mask, shape = (1, 128)\n",
            "INFO:tensorflow:  name = label_ids, shape = (1,)\n",
            "INFO:tensorflow:  name = segment_ids, shape = (1, 128)\n",
            "WARNING:tensorflow:From bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.dense instead.\n",
            "INFO:tensorflow:**** Trainable Variables ****\n",
            "INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (30522, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_12/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_13/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_14/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_15/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_16/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_17/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_18/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_19/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_20/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_21/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_22/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/encoder/layer_23/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "INFO:tensorflow:  name = output_weights:0, shape = (3, 1024)\n",
            "INFO:tensorflow:  name = output_bias:0, shape = (3,)\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:TPU job name worker\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "INFO:tensorflow:Restoring parameters from gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new/model.ckpt-199\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Init TPU system\n",
            "INFO:tensorflow:Initialized TPU in 7 seconds\n",
            "INFO:tensorflow:Starting infeed thread controller.\n",
            "INFO:tensorflow:Starting outfeed thread controller.\n",
            "INFO:tensorflow:Initialized dataset iterators in 0 seconds\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n",
            "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n",
            "INFO:tensorflow:Stop infeed thread controller\n",
            "INFO:tensorflow:Shutting down InfeedController thread.\n",
            "INFO:tensorflow:InfeedController received shutdown signal, stopping.\n",
            "INFO:tensorflow:Infeed thread finished, shutting down.\n",
            "INFO:tensorflow:infeed marked as finished\n",
            "INFO:tensorflow:Stop output thread controller\n",
            "INFO:tensorflow:Shutting down OutfeedController thread.\n",
            "INFO:tensorflow:OutfeedController received shutdown signal, stopping.\n",
            "INFO:tensorflow:Outfeed thread finished, shutting down.\n",
            "INFO:tensorflow:outfeed marked as finished\n",
            "INFO:tensorflow:Shutdown TPU system.\n",
            "INFO:tensorflow:prediction_loop marked as finished\n",
            "INFO:tensorflow:prediction_loop marked as finished\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-0uvKcDFgFl0",
        "colab_type": "code",
        "outputId": "90c810d2-2125-436e-cb5f-e478a85248bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import numpy as np\n",
        "np.argmax(all_preds[0]['probabilities'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dduj731agdEr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test = eval_examples"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NiCABpYkh3Sk",
        "colab_type": "code",
        "outputId": "56c33bab-71a5-4fbc-cdd4-abd041278e2c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "test[0].text_a"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'# mtp # meet the press how is deleting emails part of the government record different from erasing parts of a tape ? # nixon # # p2 #'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KCMQUUx_h7VL",
        "colab_type": "code",
        "outputId": "354db02d-2f72-4e06-82e9-e72d3b3e7317",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        }
      },
      "source": [
        "import numpy as np\n",
        "matrix = np.array([[0,0,0],[0,0,0]])\n",
        "for i in range(len(all_preds)):\n",
        "  gold = int(test[i].label)\n",
        "  pred = np.argmax(all_preds[i]['probabilities'])\n",
        "  if gold<2:\n",
        "    matrix[gold][2]+=1\n",
        "  if pred < 2:\n",
        "    matrix[pred][1]+=1\n",
        "    if gold == pred:\n",
        "      matrix[gold][0]+=1\n",
        "  \n",
        "print(matrix)\n",
        "a = matrix[0][0]/(matrix[0][1]+matrix[0][2]+1e-5)\n",
        "b = matrix[1][0]/(matrix[1][1]+matrix[1][2]+1e-5)\n",
        "print(\"fscore - \",a+b)\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[151 203 172]\n",
            " [ 26  32  45]]\n",
            "fscore -  0.7403290043290043\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XkH03g39jOk_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "labels_dict = [\"oppose\",\"support\",\"neutral\"]\n",
        "tweets = [\"tweet\"]+[t.text_a for t in test]\n",
        "gold_labels = [\"correct\"]+[labels_dict[int(t.label)] for t in test]\n",
        "pred_labels = [\"predicted\"]+[labels_dict[np.argmax(t['probabilities'])] for t in all_preds]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ltfw4XCF3FpT",
        "colab_type": "code",
        "outputId": "19ab68fd-8f91-4836-ff44-84d035891c4a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "tweets[0],gold_labels[0],pred_labels[0]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('tweet', 'correct', 'predicted')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wp7xSkCjlC_E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.savetxt('{}.csv'.format(TASK), [p for p in zip(tweets, gold_labels, pred_labels)], delimiter='\\t', fmt='%s')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ifLzbIKxldBj",
        "colab_type": "code",
        "outputId": "58a36b27-76c1-459f-de99-70f2ede0585e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!ls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "adc.json  bert\tgdrive\tHC.csv\tsample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yjnhmiktlgmn",
        "colab_type": "code",
        "outputId": "4f4ebb9e-58fd-488b-ad47-c13267fb5636",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv(\"{}.csv\".format(TASK),sep='\\t')\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>correct</th>\n",
              "      <th>predicted</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td># mtp # meet the press how is deleting emails ...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>jd son 78 andrew b roe ring andrew why do you ...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>the white male vote is solidly gop the black v...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>neutral</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ny investing big banker buds need to ratchet u...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>gop why should i believe you on this ? the gop...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               tweet correct predicted\n",
              "0  # mtp # meet the press how is deleting emails ...  oppose    oppose\n",
              "1  jd son 78 andrew b roe ring andrew why do you ...  oppose    oppose\n",
              "2  the white male vote is solidly gop the black v...  oppose   neutral\n",
              "3  ny investing big banker buds need to ratchet u...  oppose    oppose\n",
              "4  gop why should i believe you on this ? the gop...  oppose    oppose"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0g9zp-XjnUdn",
        "colab_type": "code",
        "outputId": "ba247aee-c22a-47cb-e6c1-0e86dfed1f3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df.tail()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>correct</th>\n",
              "      <th>predicted</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>290</th>\n",
              "      <td>hillary clinton looking forward too hearing yo...</td>\n",
              "      <td>support</td>\n",
              "      <td>support</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>291</th>\n",
              "      <td>mata hari krishna i'm loving it too ! draw tha...</td>\n",
              "      <td>neutral</td>\n",
              "      <td>neutral</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>292</th>\n",
              "      <td>finney k ca n't stand msnbc anymore , but hope...</td>\n",
              "      <td>support</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>293</th>\n",
              "      <td>hillary ca n't create jobs ! last time she had...</td>\n",
              "      <td>oppose</td>\n",
              "      <td>oppose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>294</th>\n",
              "      <td>it 's amazing to me how if you want a secure b...</td>\n",
              "      <td>neutral</td>\n",
              "      <td>neutral</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 tweet  correct predicted\n",
              "290  hillary clinton looking forward too hearing yo...  support   support\n",
              "291  mata hari krishna i'm loving it too ! draw tha...  neutral   neutral\n",
              "292  finney k ca n't stand msnbc anymore , but hope...  support    oppose\n",
              "293  hillary ca n't create jobs ! last time she had...   oppose    oppose\n",
              "294  it 's amazing to me how if you want a secure b...  neutral   neutral"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sTvkn2tInplE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "files.download('HC.csv') "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mlxsXfMzrrlt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.to_csv('gdrive/My Drive/BERT/HC.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZBIpBMuMuJcA",
        "colab_type": "code",
        "outputId": "1d5d2bc0-8f5e-4714-eae2-94a6a943ad57",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 131
        }
      },
      "source": [
        "SAVE_DATA_DIR = 'gdrive/My Drive/BERT/"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "SyntaxError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-51-e0d139c8d2e3>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    SAVE_DATA_DIR = 'gdrive/My Drive/BERT/\u001b[0m\n\u001b[0m                                          ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m EOL while scanning string literal\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xt73aEtgutUy",
        "colab_type": "code",
        "outputId": "ec3b4ead-e953-4006-e9d0-99afea4c05a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 551
        }
      },
      "source": [
        "df.to_csv('gdrive/My Drive/BERT/')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "IsADirectoryError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mIsADirectoryError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-32-f115bede8bb0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'gdrive/My Drive/BERT/'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36mto_csv\u001b[0;34m(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, compression, quoting, quotechar, line_terminator, chunksize, tupleize_cols, date_format, doublequote, escapechar, decimal)\u001b[0m\n\u001b[1;32m   1743\u001b[0m                                  \u001b[0mdoublequote\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdoublequote\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1744\u001b[0m                                  escapechar=escapechar, decimal=decimal)\n\u001b[0;32m-> 1745\u001b[0;31m         \u001b[0mformatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1746\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1747\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mpath_or_buf\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/formats/csvs.py\u001b[0m in \u001b[0;36msave\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    154\u001b[0m             f, handles = _get_handle(self.path_or_buf, self.mode,\n\u001b[1;32m    155\u001b[0m                                      \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 156\u001b[0;31m                                      compression=self.compression)\n\u001b[0m\u001b[1;32m    157\u001b[0m             \u001b[0mclose\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/common.py\u001b[0m in \u001b[0;36m_get_handle\u001b[0;34m(path_or_buf, mode, encoding, compression, memory_map, is_text)\u001b[0m\n\u001b[1;32m    398\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    399\u001b[0m             \u001b[0;31m# Python 3 and encoding\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 400\u001b[0;31m             \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    401\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mis_text\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    402\u001b[0m             \u001b[0;31m# Python 3 and no explicit encoding\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mIsADirectoryError\u001b[0m: [Errno 21] Is a directory: 'gdrive/My Drive/BERT/'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0jh5x7s7vzfo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}