{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Copy of BERT FineTuning with Cloud TPU: Sentence and Sentence-Pair Classification Tasks", "version": "0.3.2", "provenance": [], "collapsed_sections": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "accelerator": "TPU" }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "rkTLZ3I4_7c_", "colab_type": "text" }, "source": [ "# BERT finetuning tasks in 5 minutes with Cloud TPU\n", "\n", "\n", " \n", " \n", "
\n", " Run in Google Colab\n", " \n", " View source on GitHub\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "1wtjs1QDb3DX", "colab_type": "text" }, "source": [ "**BERT**, or **B**idirectional **E**mbedding **R**epresentations from **T**ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.\n", "\n", "This Colab demonstates using a free Colab Cloud TPU to fine-tune sentence and sentence-pair classification tasks built on top of pretrained BERT models.\n", "\n", "**Note:** You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud \n", "Storage) bucket for this Colab to run.\n", "\n", "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) for how to create GCP account and GCS bucket. You have [$300 free credit](https://cloud.google.com/free/) to get started with any GCP product. You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs.\n", "\n", "Once you finish the setup, let's start!" ] }, { "cell_type": "markdown", "metadata": { "id": "ycHMh-bhC-vX", "colab_type": "text" }, "source": [ "**Firstly**, we need to set up Colab TPU running environment, verify a TPU device is succesfully connected and upload credentials to TPU for GCS bucket usage." ] }, { "cell_type": "code", "metadata": { "id": "191zq3ZErihP", "colab_type": "code", "outputId": "1a6a8e2d-a095-4732-eed5-917acf31b1c6", "colab": { "base_uri": "https://localhost:8080/", "height": 364 } }, "source": [ "import datetime\n", "import json\n", "import os\n", "import pprint\n", "import random\n", "import string\n", "import sys\n", "import tensorflow as tf\n", "\n", "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n", "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n", "print('TPU address is', TPU_ADDRESS)\n", "\n", "from google.colab import auth\n", "auth.authenticate_user()\n", "with tf.Session(TPU_ADDRESS) as session:\n", " print('TPU devices:')\n", " pprint.pprint(session.list_devices())\n", "\n", " # Upload credentials to TPU.\n", " with open('/content/adc.json', 'r') as f:\n", " auth_info = json.load(f)\n", " tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n", " # Now credentials are set for all future sessions on this TPU." ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "TPU address is grpc://10.77.133.178:8470\n", "TPU devices:\n", "[_DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:CPU:0, CPU, -1, 11193819257324014272),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 11743471906924308223),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 8817390715541045037),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 13647766437076739427),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 12291117123528086541),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 5167520003982709723),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 1806457377803209548),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 10367945529533593525),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 11821730786090633668),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 17853583825165986593),\n", " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 11451074782555305885)]\n", "\n", "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n", "For more information, please see:\n", " * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n", " * https://github.com/tensorflow/addons\n", "If you depend on functionality not listed there, please file an issue.\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "HUBP35oCDmbF", "colab_type": "text" }, "source": [ "**Secondly**, prepare and import BERT modules." ] }, { "cell_type": "code", "metadata": { "id": "tEXK_HWODp60", "colab_type": "code", "colab": {} }, "source": [ "!rm -rf bert" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7wzwke0sxS6W", "colab_type": "code", "colab": {} }, "source": [ "import sys\n", "\n", "!test -d bert || git clone https://github.com/shalmolighosh/bert/\n", "if not 'bert' in sys.path:\n", " sys.path += ['bert']" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "y5tDCk6tB-ee", "colab_type": "code", "colab": {} }, "source": [ "#!cat bert/run_classifier.py | grep -C 10 _read_csv" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "JXoAweIA8kYN", "colab_type": "code", "outputId": "3d6f6154-4da2-4e8e-9e2d-1f4edcec51e1", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "from google.colab import drive\n", "drive.mount('/content/gdrive')" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "O3PXnf6E9b5c", "colab_type": "code", "outputId": "87fe0f1a-516b-4669-cb80-251bd937d5db", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "!ls gdrive/My\\ Drive/BERT/Data" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "Atheism CC FM HC LA\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "RRu1aKO1D7-Z", "colab_type": "text" }, "source": [ "**Thirdly**, prepare for training:\n", "\n", "* Specify task and download training data.\n", "* Specify BERT pretrained model\n", "* Specify GS bucket, create output directory for model checkpoints and eval results.\n", "\n" ] }, { "cell_type": "code", "metadata": { "id": "tYkaAlJNfhul", "colab_type": "code", "outputId": "e1a98277-8267-4fea-cc83-d528bbcace3c", "cellView": "both", "colab": { "base_uri": "https://localhost:8080/", "height": 191 } }, "source": [ "TASK = 'HC' #@param {type:\"string\"}\n", "assert TASK in ('MRPC', 'CoLA','Atheism','CC','HC','LA','FM','ALL'), 'Only (MRPC, CoLA, Sem) are demonstrated here.'\n", "# Download glue data.\n", "if TASK=='MRPC' or TASK == 'CoLA':\n", " ! test -d download_glue_repo || git clone https://gist.github.com/60c2bdb54d156a41194446737ce03e2e.git download_glue_repo\n", " !python download_glue_repo/download_glue_data.py --data_dir='glue_data' --tasks=$TASK\n", " TASK_DATA_DIR = 'glue_data/' + TASK\n", "\n", "elif TASK!='ALL':\n", " TASK_DATA_DIR = 'gdrive/My\\ Drive/BERT/Data/' + TASK\n", "\n", "else:\n", " TASK_DATA_DIR = 'gdrive/My\\ Drive/BERT/Data/'\n", " \n", " \n", "print('***** Task data directory: {} *****'.format(TASK_DATA_DIR))\n", "!ls $TASK_DATA_DIR\n", "\n", "# Available pretrained model checkpoints:\n", "# uncased_L-12_H-768_A-12: uncased BERT base model\n", "# uncased_L-24_H-1024_A-16: uncased BERT large model\n", "# cased_L-12_H-768_A-12: cased BERT large model\n", "BERT_MODEL = 'uncased_L-24_H-1024_A-16' #@param {type:\"string\"}\n", "BERT_PRETRAINED_DIR = 'gs://cloud-tpu-checkpoints/bert/' + BERT_MODEL\n", "print('***** BERT pretrained directory: {} *****'.format(BERT_PRETRAINED_DIR))\n", "!gsutil ls $BERT_PRETRAINED_DIR\n", "\n", "BUCKET = 'bert-large-pair' #@param {type:\"string\"}\n", "assert BUCKET, 'Must specify an existing GCS bucket name'\n", "OUTPUT_DIR = 'gs://{}/bert/models/{}/{}_new'.format(BUCKET,BERT_MODEL ,TASK)\n", "tf.gfile.MakeDirs(OUTPUT_DIR)\n", "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "***** Task data directory: gdrive/My\\ Drive/BERT/Data/HC *****\n", "test_preprocessed.csv train_preprocessed.csv\n", "***** BERT pretrained directory: gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16 *****\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_config.json\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.data-00000-of-00001\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.index\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/bert_model.ckpt.meta\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/checkpoint\n", "gs://cloud-tpu-checkpoints/bert/uncased_L-24_H-1024_A-16/vocab.txt\n", "***** Model output directory: gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new *****\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "Qpn1MM6dd7YS", "colab_type": "code", "colab": {} }, "source": [ "#!gsutil cp gs://bert-final/bert/models/Atheism/* gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/Atheism" ], "execution_count": 0, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Hcpfl4N2EdOk", "colab_type": "text" }, "source": [ "**Now, let's play!**" ] }, { "cell_type": "code", "metadata": { "id": "uu2dQ_TId-uH", "colab_type": "code", "outputId": "0a739553-e406-4059-f3ca-406c3a92d031", "cellView": "both", "colab": { "base_uri": "https://localhost:8080/", "height": 298 } }, "source": [ "# Setup task specific model and TPU running config.\n", "\n", "import modeling\n", "import optimization\n", "import run_classifier\n", "import tokenization\n", "\n", "if TASK!='ALL':\n", " TASK_DATA_DIR = 'gdrive/My Drive/BERT/Data/' + TASK\n", "else:\n", " TASK_DATA_DIR = 'gdrive/My Drive/BERT/Data/'\n", " \n", "# Model Hyper Parameters\n", "TRAIN_BATCH_SIZE = 32\n", "EVAL_BATCH_SIZE = 8\n", "LEARNING_RATE = 2e-5\n", "NUM_TRAIN_EPOCHS = \"11\" #@param {type:\"string\"}\n", "NUM_TRAIN_EPOCHS = int(NUM_TRAIN_EPOCHS)\n", "WARMUP_PROPORTION = 0.1\n", "MAX_SEQ_LENGTH = 128\n", "# Model configs\n", "SAVE_CHECKPOINTS_STEPS = 1000\n", "ITERATIONS_PER_LOOP = 1000\n", "NUM_TPU_CORES = 8\n", "VOCAB_FILE = os.path.join(BERT_PRETRAINED_DIR, 'vocab.txt')\n", "CONFIG_FILE = os.path.join(BERT_PRETRAINED_DIR, 'bert_config.json')\n", "INIT_CHECKPOINT = os.path.join(BERT_PRETRAINED_DIR, 'bert_model.ckpt')\n", "DO_LOWER_CASE = BERT_MODEL.startswith('uncased')\n", "\n", "processors = {\n", " \"cola\": run_classifier.ColaProcessor,\n", " \"mnli\": run_classifier.MnliProcessor,\n", " \"mrpc\": run_classifier.MrpcProcessor,\n", " \"hc\" : run_classifier.SemProcessor,\n", " \"atheism\" : run_classifier.SemProcessor,\n", " \"fm\" : run_classifier.SemProcessor,\n", " \"cc\" : run_classifier.SemProcessor,\n", " \"la\" : run_classifier.SemProcessor,\n", " \"all\" : run_classifier.SemProcessor\n", "}\n", "print(processors[TASK.lower()])\n", "\n", "tokenizer = tokenization.FullTokenizer(vocab_file=VOCAB_FILE, do_lower_case=DO_LOWER_CASE)\n", "\n", "tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n", "run_config = tf.contrib.tpu.RunConfig(\n", " cluster=tpu_cluster_resolver,\n", " model_dir=OUTPUT_DIR,\n", " save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,\n", " tpu_config=tf.contrib.tpu.TPUConfig(\n", " iterations_per_loop=ITERATIONS_PER_LOOP,\n", " num_shards=NUM_TPU_CORES,\n", " per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))\n", "\n", "if TASK == 'ALL':\n", " train_examples = []\n", " full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n", " for key,value in full_forms.items():\n", " processor = run_classifier.SemProcessor(use_pair=True, topic = value)\n", " label_list = processor.get_labels()\n", " train_examples += processor.get_train_examples(TASK_DATA_DIR+key) \n", "\n", "else:\n", " full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n", " processor = processors[TASK.lower()](use_pair=False,\\\n", " topic=full_forms[TASK])\n", " label_list = processor.get_labels()\n", " train_examples = processor.get_train_examples(TASK_DATA_DIR)\n", "\n", "\n", "print(\"Number of train examples :\",len(train_examples))\n", " \n", "num_train_steps = int(\n", " len(train_examples) / TRAIN_BATCH_SIZE * NUM_TRAIN_EPOCHS)\n", "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)\n", "\n", "model_fn = run_classifier.model_fn_builder(\n", " bert_config=modeling.BertConfig.from_json_file(CONFIG_FILE),\n", " num_labels=len(label_list),\n", " init_checkpoint=INIT_CHECKPOINT,\n", " learning_rate=LEARNING_RATE,\n", " num_train_steps=num_train_steps,\n", " num_warmup_steps=num_warmup_steps,\n", " use_tpu=True,\n", " use_one_hot_embeddings=True)\n", "\n", "estimator = tf.contrib.tpu.TPUEstimator(\n", " use_tpu=True,\n", " model_fn=model_fn,\n", " config=run_config,\n", " train_batch_size=TRAIN_BATCH_SIZE,\n", " eval_batch_size=EVAL_BATCH_SIZE)" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "\n", "Number of train examples : 639\n", "WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x7fcce37cf9d8>) includes params argument, but params are not passed to Estimator.\n", "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n", "cluster_def {\n", " job {\n", " name: \"worker\"\n", " tasks {\n", " key: 0\n", " value: \"10.77.133.178:8470\"\n", " }\n", " }\n", "}\n", ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.77.133.178:8470', '_evaluation_master': 'grpc://10.77.133.178:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': }\n", "INFO:tensorflow:_TPUContext: eval_on_tpu True\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "HCHohyPJFtmc", "colab_type": "code", "colab": {} }, "source": [ "import csv\n", "def read_csv(input_file, quotechar=None):\n", " \"\"\"Reads a tab separated value file.\"\"\"\n", " with tf.gfile.Open(input_file, \"r\") as f:\n", " reader = reader = csv.reader(f)\n", " lines = []\n", " for line in reader:\n", " if sys.version_info[0]==2:\n", " line = list(unicode(cell, 'utf-8') for cell in line)\n", " lines.append(line)\n", " return lines\n" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "kqPahgY9GMQK", "colab_type": "code", "colab": {} }, "source": [ "#lines = read_csv(TASK_DATA_DIR+'/train_preprocessed.csv')" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "TTvlyZ7KGhMM", "colab_type": "code", "colab": {} }, "source": [ "#lines" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "5U_c8s2AvhgL", "colab_type": "code", "outputId": "450c3bb7-e7c4-47d5-9eaf-a5a66dc61aa6", "colab": { "base_uri": "https://localhost:8080/", "height": 766 } }, "source": [ "# Train the model.\n", "print('MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...')\n", "train_features = run_classifier.convert_examples_to_features(\n", " train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n", "print('***** Started training at {} *****'.format(datetime.datetime.now()))\n", "print(' Num examples = {}'.format(len(train_examples)))\n", "print(' Batch size = {}'.format(TRAIN_BATCH_SIZE))\n", "tf.logging.info(\" Num steps = %d\", num_train_steps)\n", "train_input_fn = run_classifier.input_fn_builder(\n", " features=train_features,\n", " seq_length=MAX_SEQ_LENGTH,\n", " is_training=True,\n", " drop_remainder=True)\n", "#estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n", "print('***** Finished training at {} *****'.format(datetime.datetime.now()))" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...\n", "INFO:tensorflow:Writing example 0 of 639\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: train-0\n", "INFO:tensorflow:tokens: [CLS] rt gunn jessica because i want young american women to be able to be proud of the 1st woman president [SEP]\n", "INFO:tensorflow:input_ids: 101 19387 22079 8201 2138 1045 2215 2402 2137 2308 2000 2022 2583 2000 2022 7098 1997 1996 3083 2450 2343 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 1 (id = 1)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: train-1\n", "INFO:tensorflow:tokens: [CLS] chris 1791 news buster ##s too years ago # hillary # never answered whether she used private email # liberal # media passed on reporting [SEP]\n", "INFO:tensorflow:input_ids: 101 3782 14362 2739 18396 2015 2205 2086 3283 1001 18520 1001 2196 4660 3251 2016 2109 2797 10373 1001 4314 1001 2865 2979 2006 7316 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: train-2\n", "INFO:tensorflow:tokens: [CLS] d new ha ##user take that , marco rub ##io go ##p cannot pan ##der to the latino vote while pushing policies that hurt the community most [SEP]\n", "INFO:tensorflow:input_ids: 101 1040 2047 5292 20330 2202 2008 1010 8879 14548 3695 2175 2361 3685 6090 4063 2000 1996 7402 3789 2096 6183 6043 2008 3480 1996 2451 2087 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 2 (id = 2)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: train-3\n", "INFO:tensorflow:tokens: [CLS] state dept am ##b cathy russell adam smith usa hillary clinton hillary for i a barack obama the whole # world # is supporting you [SEP]\n", "INFO:tensorflow:input_ids: 101 2110 29466 2572 2497 18305 5735 4205 3044 3915 18520 7207 18520 2005 1045 1037 13857 8112 1996 2878 1001 2088 1001 2003 4637 2017 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 1 (id = 1)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: train-4\n", "INFO:tensorflow:tokens: [CLS] lets remember # dick cheney # is an un ##ind ##ic ##ted war criminal before we start yelling # ben ##gh ##azi # day after day will we ever see justice ? [SEP]\n", "INFO:tensorflow:input_ids: 101 11082 3342 1001 5980 23745 1001 2003 2019 4895 22254 2594 3064 2162 4735 2077 2057 2707 13175 1001 3841 5603 16103 1001 2154 2044 2154 2097 2057 2412 2156 3425 1029 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 2 (id = 2)\n", "***** Started training at 2019-04-13 17:27:42.745922 *****\n", " Num examples = 639\n", " Batch size = 32\n", "INFO:tensorflow: Num steps = 199\n", "***** Finished training at 2019-04-13 17:27:42.748669 *****\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "vA-Vf52AGI2r", "colab_type": "code", "outputId": "89a8ab47-3156-434b-af2a-3052a5175a7d", "colab": { "base_uri": "https://localhost:8080/", "height": 263 } }, "source": [ "estimator = tf.contrib.tpu.TPUEstimator(\n", " use_tpu=True,\n", " model_fn=model_fn,\n", " config=run_config,\n", " train_batch_size=TRAIN_BATCH_SIZE,\n", " eval_batch_size=EVAL_BATCH_SIZE,\n", " predict_batch_size = EVAL_BATCH_SIZE)" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x7fcce37cf9d8>) includes params argument, but params are not passed to Estimator.\n", "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n", "cluster_def {\n", " job {\n", " name: \"worker\"\n", " tasks {\n", " key: 0\n", " value: \"10.77.133.178:8470\"\n", " }\n", " }\n", "}\n", ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.77.133.178:8470', '_evaluation_master': 'grpc://10.77.133.178:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': }\n", "INFO:tensorflow:_TPUContext: eval_on_tpu True\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "eoXRtSPZvdiS", "colab_type": "code", "outputId": "bb64befc-a858-4d12-b9e2-497b8a9ac132", "colab": { "base_uri": "https://localhost:8080/", "height": 732 } }, "source": [ "# Eval the model.\n", "if TASK != 'ALL':\n", " eval_examples = processor.get_dev_examples(TASK_DATA_DIR)\n", " \n", "else:\n", " eval_examples = []\n", " full_forms = {'HC' : 'hillary clinton', 'CC' : 'climate change is a concern','Atheism' : 'Atheism', 'LA' : 'Legalisation of Abortion', 'FM' : 'Feminist Movement'}\n", " for key,value in full_forms.items():\n", " processor = run_classifier.SemProcessor(use_pair=True, topic = value)\n", " eval_examples += processor.get_dev_examples(TASK_DATA_DIR+key) \n", "\n", "eval_features = run_classifier.convert_examples_to_features(\n", " eval_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n", "print('***** Started evaluation at {} *****'.format(datetime.datetime.now()))\n", "print(' Num examples = {}'.format(len(eval_examples)))\n", "print(' Batch size = {}'.format(EVAL_BATCH_SIZE))\n", "# Eval will be slightly WRONG on the TPU because it will truncate\n", "# the last batch.\n", "eval_steps = int(len(eval_examples) / EVAL_BATCH_SIZE)\n", "eval_input_fn = run_classifier.input_fn_builder(\n", " features=eval_features,\n", " seq_length=MAX_SEQ_LENGTH,\n", " is_training=False,\n", " drop_remainder=True)\n", "#result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)\n", "print('***** Finished evaluation at {} *****'.format(datetime.datetime.now()))\n", "#output_eval_file = os.path.join(OUTPUT_DIR, \"eval_results.txt\")\n", "#with tf.gfile.GFile(output_eval_file, \"w\") as writer:\n", "# print(\"***** Eval results *****\")\n", "# for key in sorted(result.keys()):\n", "# print(' {} = {}'.format(key, str(result[key])))\n", "# writer.write(\"%s = %s\\n\" % (key, str(result[key])))\n" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "INFO:tensorflow:Writing example 0 of 295\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: dev-0\n", "INFO:tensorflow:tokens: [CLS] # mt ##p # meet the press how is del ##eti ##ng emails part of the government record different from eras ##ing parts of a tape ? # nixon # # p ##2 # [SEP]\n", "INFO:tensorflow:input_ids: 101 1001 11047 2361 1001 3113 1996 2811 2129 2003 3972 20624 3070 22028 2112 1997 1996 2231 2501 2367 2013 28500 2075 3033 1997 1037 6823 1029 1001 11296 1001 1001 1052 2475 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: dev-1\n", "INFO:tensorflow:tokens: [CLS] jd son 78 andrew b roe ring andrew why do you care about what i think ? i did not realize that i was this important sir t ##wee ##t andrew is a paid troll [SEP]\n", "INFO:tensorflow:input_ids: 101 26219 2365 6275 4080 1038 20944 3614 4080 2339 2079 2017 2729 2055 2054 1045 2228 1029 1045 2106 2025 5382 2008 1045 2001 2023 2590 2909 1056 28394 2102 4080 2003 1037 3825 18792 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: dev-2\n", "INFO:tensorflow:tokens: [CLS] the white male vote is solid ##ly go ##p the black vote is solid ##ly them that leaves white females and brown people # feel the bern # [SEP]\n", "INFO:tensorflow:input_ids: 101 1996 2317 3287 3789 2003 5024 2135 2175 2361 1996 2304 3789 2003 5024 2135 2068 2008 3727 2317 3801 1998 2829 2111 1001 2514 1996 16595 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: dev-3\n", "INFO:tensorflow:tokens: [CLS] ny investing big banker buds need to rat ##chet up their hillary cares about the little people propaganda [SEP]\n", "INFO:tensorflow:input_ids: 101 6396 19920 2502 13448 26734 2342 2000 9350 20318 2039 2037 18520 14977 2055 1996 2210 2111 10398 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "INFO:tensorflow:*** Example ***\n", "INFO:tensorflow:guid: dev-4\n", "INFO:tensorflow:tokens: [CLS] go ##p why should i believe you on this ? the go ##p leaders in congress wo n ' t fight obama now ! ! ! # t cot # [SEP]\n", "INFO:tensorflow:input_ids: 101 2175 2361 2339 2323 1045 2903 2017 2006 2023 1029 1996 2175 2361 4177 1999 3519 24185 1050 1005 1056 2954 8112 2085 999 999 999 1001 1056 26046 1001 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", "INFO:tensorflow:label: 0 (id = 0)\n", "***** Started evaluation at 2019-04-13 17:28:33.466208 *****\n", " Num examples = 295\n", " Batch size = 8\n", "***** Finished evaluation at 2019-04-13 17:28:33.467656 *****\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "gW32URRgenOv", "colab_type": "code", "colab": {} }, "source": [ "#?? estimator" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "bjVueAUieoO_", "colab_type": "code", "colab": {} }, "source": [ "preds = estimator.predict(\n", " input_fn=eval_input_fn)" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ULIjFr-YgAP_", "colab_type": "code", "outputId": "1f6f495a-6521-459e-bb26-ec8510c3a039", "colab": { "base_uri": "https://localhost:8080/", "height": 9102 } }, "source": [ "all_preds = []\n", "for pred in preds:\n", " all_preds.append(pred)\n" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "INFO:tensorflow:Querying Tensorflow master (grpc://10.77.133.178:8470) for TPU system metadata.\n", "INFO:tensorflow:Found TPU system:\n", "INFO:tensorflow:*** Num TPU Cores: 8\n", "INFO:tensorflow:*** Num TPU Workers: 1\n", "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 11193819257324014272)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 11743471906924308223)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 8817390715541045037)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 13647766437076739427)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 12291117123528086541)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 5167520003982709723)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 1806457377803209548)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 10367945529533593525)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 11821730786090633668)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 17853583825165986593)\n", "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 11451074782555305885)\n", "INFO:tensorflow:Calling model_fn.\n", "INFO:tensorflow:*** Features ***\n", "INFO:tensorflow: name = input_ids, shape = (1, 128)\n", "INFO:tensorflow: name = input_mask, shape = (1, 128)\n", "INFO:tensorflow: name = label_ids, shape = (1,)\n", "INFO:tensorflow: name = segment_ids, shape = (1, 128)\n", "WARNING:tensorflow:From bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use keras.layers.dense instead.\n", "INFO:tensorflow:**** Trainable Variables ****\n", "INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (30522, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_12/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_13/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_14/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_15/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_16/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_17/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_18/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_19/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_20/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_21/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_22/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/encoder/layer_23/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n", "INFO:tensorflow: name = output_weights:0, shape = (3, 1024)\n", "INFO:tensorflow: name = output_bias:0, shape = (3,)\n", "INFO:tensorflow:Done calling model_fn.\n", "INFO:tensorflow:TPU job name worker\n", "INFO:tensorflow:Graph was finalized.\n", "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use standard file APIs to check for files with this prefix.\n", "INFO:tensorflow:Restoring parameters from gs://bert-large-pair/bert/models/uncased_L-24_H-1024_A-16/HC_new/model.ckpt-199\n", "INFO:tensorflow:Running local_init_op.\n", "INFO:tensorflow:Done running local_init_op.\n", "INFO:tensorflow:Init TPU system\n", "INFO:tensorflow:Initialized TPU in 7 seconds\n", "INFO:tensorflow:Starting infeed thread controller.\n", "INFO:tensorflow:Starting outfeed thread controller.\n", "INFO:tensorflow:Initialized dataset iterators in 0 seconds\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n", "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n", "INFO:tensorflow:Stop infeed thread controller\n", "INFO:tensorflow:Shutting down InfeedController thread.\n", "INFO:tensorflow:InfeedController received shutdown signal, stopping.\n", "INFO:tensorflow:Infeed thread finished, shutting down.\n", "INFO:tensorflow:infeed marked as finished\n", "INFO:tensorflow:Stop output thread controller\n", "INFO:tensorflow:Shutting down OutfeedController thread.\n", "INFO:tensorflow:OutfeedController received shutdown signal, stopping.\n", "INFO:tensorflow:Outfeed thread finished, shutting down.\n", "INFO:tensorflow:outfeed marked as finished\n", "INFO:tensorflow:Shutdown TPU system.\n", "INFO:tensorflow:prediction_loop marked as finished\n", "INFO:tensorflow:prediction_loop marked as finished\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "-0uvKcDFgFl0", "colab_type": "code", "outputId": "90c810d2-2125-436e-cb5f-e478a85248bb", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "import numpy as np\n", "np.argmax(all_preds[0]['probabilities'])" ], "execution_count": 0, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "0" ] }, "metadata": { "tags": [] }, "execution_count": 33 } ] }, { "cell_type": "code", "metadata": { "id": "Dduj731agdEr", "colab_type": "code", "colab": {} }, "source": [ "test = eval_examples" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "NiCABpYkh3Sk", "colab_type": "code", "outputId": "56c33bab-71a5-4fbc-cdd4-abd041278e2c", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "test[0].text_a" ], "execution_count": 0, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'# mtp # meet the press how is deleting emails part of the government record different from erasing parts of a tape ? # nixon # # p2 #'" ] }, "metadata": { "tags": [] }, "execution_count": 22 } ] }, { "cell_type": "code", "metadata": { "id": "KCMQUUx_h7VL", "colab_type": "code", "outputId": "354db02d-2f72-4e06-82e9-e72d3b3e7317", "colab": { "base_uri": "https://localhost:8080/", "height": 69 } }, "source": [ "import numpy as np\n", "matrix = np.array([[0,0,0],[0,0,0]])\n", "for i in range(len(all_preds)):\n", " gold = int(test[i].label)\n", " pred = np.argmax(all_preds[i]['probabilities'])\n", " if gold<2:\n", " matrix[gold][2]+=1\n", " if pred < 2:\n", " matrix[pred][1]+=1\n", " if gold == pred:\n", " matrix[gold][0]+=1\n", " \n", "print(matrix)\n", "a = matrix[0][0]/(matrix[0][1]+matrix[0][2]+1e-5)\n", "b = matrix[1][0]/(matrix[1][1]+matrix[1][2]+1e-5)\n", "print(\"fscore - \",a+b)\n", " " ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "[[151 203 172]\n", " [ 26 32 45]]\n", "fscore - 0.7403290043290043\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "XkH03g39jOk_", "colab_type": "code", "colab": {} }, "source": [ "labels_dict = [\"oppose\",\"support\",\"neutral\"]\n", "tweets = [\"tweet\"]+[t.text_a for t in test]\n", "gold_labels = [\"correct\"]+[labels_dict[int(t.label)] for t in test]\n", "pred_labels = [\"predicted\"]+[labels_dict[np.argmax(t['probabilities'])] for t in all_preds]" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Ltfw4XCF3FpT", "colab_type": "code", "outputId": "19ab68fd-8f91-4836-ff44-84d035891c4a", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "tweets[0],gold_labels[0],pred_labels[0]" ], "execution_count": 0, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "('tweet', 'correct', 'predicted')" ] }, "metadata": { "tags": [] }, "execution_count": 25 } ] }, { "cell_type": "code", "metadata": { "id": "Wp7xSkCjlC_E", "colab_type": "code", "colab": {} }, "source": [ "np.savetxt('{}.csv'.format(TASK), [p for p in zip(tweets, gold_labels, pred_labels)], delimiter='\\t', fmt='%s')" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ifLzbIKxldBj", "colab_type": "code", "outputId": "58a36b27-76c1-459f-de99-70f2ede0585e", "colab": { "base_uri": "https://localhost:8080/", "height": 34 } }, "source": [ "!ls" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "adc.json bert\tgdrive\tHC.csv\tsample_data\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "yjnhmiktlgmn", "colab_type": "code", "outputId": "4f4ebb9e-58fd-488b-ad47-c13267fb5636", "colab": { "base_uri": "https://localhost:8080/", "height": 206 } }, "source": [ "import pandas as pd\n", "df = pd.read_csv(\"{}.csv\".format(TASK),sep='\\t')\n", "df.head()" ], "execution_count": 0, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
tweetcorrectpredicted
0# mtp # meet the press how is deleting emails ...opposeoppose
1jd son 78 andrew b roe ring andrew why do you ...opposeoppose
2the white male vote is solidly gop the black v...opposeneutral
3ny investing big banker buds need to ratchet u...opposeoppose
4gop why should i believe you on this ? the gop...opposeoppose
\n", "
" ], "text/plain": [ " tweet correct predicted\n", "0 # mtp # meet the press how is deleting emails ... oppose oppose\n", "1 jd son 78 andrew b roe ring andrew why do you ... oppose oppose\n", "2 the white male vote is solidly gop the black v... oppose neutral\n", "3 ny investing big banker buds need to ratchet u... oppose oppose\n", "4 gop why should i believe you on this ? the gop... oppose oppose" ] }, "metadata": { "tags": [] }, "execution_count": 28 } ] }, { "cell_type": "code", "metadata": { "id": "0g9zp-XjnUdn", "colab_type": "code", "outputId": "ba247aee-c22a-47cb-e6c1-0e86dfed1f3b", "colab": { "base_uri": "https://localhost:8080/", "height": 206 } }, "source": [ "df.tail()" ], "execution_count": 0, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
tweetcorrectpredicted
290hillary clinton looking forward too hearing yo...supportsupport
291mata hari krishna i'm loving it too ! draw tha...neutralneutral
292finney k ca n't stand msnbc anymore , but hope...supportoppose
293hillary ca n't create jobs ! last time she had...opposeoppose
294it 's amazing to me how if you want a secure b...neutralneutral
\n", "
" ], "text/plain": [ " tweet correct predicted\n", "290 hillary clinton looking forward too hearing yo... support support\n", "291 mata hari krishna i'm loving it too ! draw tha... neutral neutral\n", "292 finney k ca n't stand msnbc anymore , but hope... support oppose\n", "293 hillary ca n't create jobs ! last time she had... oppose oppose\n", "294 it 's amazing to me how if you want a secure b... neutral neutral" ] }, "metadata": { "tags": [] }, "execution_count": 29 } ] }, { "cell_type": "code", "metadata": { "id": "sTvkn2tInplE", "colab_type": "code", "colab": {} }, "source": [ "from google.colab import files\n", "files.download('HC.csv') " ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "mlxsXfMzrrlt", "colab_type": "code", "colab": {} }, "source": [ "df.to_csv('gdrive/My Drive/BERT/HC.csv')" ], "execution_count": 0, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ZBIpBMuMuJcA", "colab_type": "code", "outputId": "1d5d2bc0-8f5e-4714-eae2-94a6a943ad57", "colab": { "base_uri": "https://localhost:8080/", "height": 131 } }, "source": [ "SAVE_DATA_DIR = 'gdrive/My Drive/BERT/" ], "execution_count": 0, "outputs": [ { "output_type": "error", "ename": "SyntaxError", "evalue": "ignored", "traceback": [ "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m SAVE_DATA_DIR = 'gdrive/My Drive/BERT/\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m EOL while scanning string literal\n" ] } ] }, { "cell_type": "code", "metadata": { "id": "Xt73aEtgutUy", "colab_type": "code", "outputId": "ec3b4ead-e953-4006-e9d0-99afea4c05a3", "colab": { "base_uri": "https://localhost:8080/", "height": 551 } }, "source": [ "df.to_csv('gdrive/My Drive/BERT/')" ], "execution_count": 0, "outputs": [ { "output_type": "error", "ename": "IsADirectoryError", "evalue": "ignored", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mIsADirectoryError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'gdrive/My Drive/BERT/'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36mto_csv\u001b[0;34m(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, compression, quoting, quotechar, line_terminator, chunksize, tupleize_cols, date_format, doublequote, escapechar, decimal)\u001b[0m\n\u001b[1;32m 1743\u001b[0m \u001b[0mdoublequote\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdoublequote\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1744\u001b[0m escapechar=escapechar, decimal=decimal)\n\u001b[0;32m-> 1745\u001b[0;31m \u001b[0mformatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1746\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1747\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mpath_or_buf\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/formats/csvs.py\u001b[0m in \u001b[0;36msave\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 154\u001b[0m f, handles = _get_handle(self.path_or_buf, self.mode,\n\u001b[1;32m 155\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 156\u001b[0;31m compression=self.compression)\n\u001b[0m\u001b[1;32m 157\u001b[0m \u001b[0mclose\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 158\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/common.py\u001b[0m in \u001b[0;36m_get_handle\u001b[0;34m(path_or_buf, mode, encoding, compression, memory_map, is_text)\u001b[0m\n\u001b[1;32m 398\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 399\u001b[0m \u001b[0;31m# Python 3 and encoding\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 400\u001b[0;31m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 401\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mis_text\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 402\u001b[0m \u001b[0;31m# Python 3 and no explicit encoding\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mIsADirectoryError\u001b[0m: [Errno 21] Is a directory: 'gdrive/My Drive/BERT/'" ] } ] }, { "cell_type": "code", "metadata": { "id": "0jh5x7s7vzfo", "colab_type": "code", "colab": {} }, "source": [ "" ], "execution_count": 0, "outputs": [] } ] }