{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 交差エントロピーの計算\n",
    "\n",
    "ディープラーニングの分類問題における損失関数として一般的に使用されている**交差エントロピー**に関していろいろな解説記事を読んでいると諸説あり、Chainerで実装されているのはどちらかを確認します。\n",
    "\n",
    "$\\displaystyle \\mathcal{L} = - \\dfrac{1}{N}\\sum_{n} \\sum_{k} t_{nk} \\log y_{nk}$\n",
    "\n",
    "もしくは\n",
    "\n",
    "$\\displaystyle \\mathcal{L} = - \\dfrac{1}{N} \\dfrac{1}{K} \\sum_{n}  \\sum_{k} \\left\\{ t_{nk} \\log y_{nk} + (1 - t_{nk}) \\log (1 - y_{nk}) \\right\\}$\n",
    "\n",
    "となっている２パターンがありました。この違いは、教師データとして与えられた正解ラベル以外の要素を考慮するかしないかになります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chainerで損失関数の計算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import chainer.functions as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = np.array([0], 'i')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.array([[0.7, 0.2, 0.1]], 'f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "variable(0.76794964)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = F.softmax_cross_entropy(y, t)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 検証"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "variable([[0.46396342, 0.28140804, 0.25462854]])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_softmax = F.softmax(y)\n",
    "y_softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7679495660705155"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "- np.log(0.46396342)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3304615927872412"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "- np.log(1 - 0.28140804)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2938725808374165"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-np.log(1 - 0.25462854)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3922837396951733"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "- np.log(0.46396342) - np.log(1 - 0.28140804) -np.log(1 - 0.25462854)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**検証結果**\n",
    "\n",
    "Chainerでのクロスエントロピーは、\n",
    "\n",
    "$\\displaystyle \\mathcal{L} = - \\dfrac{1}{N}\\sum_{n} \\sum_{k} t_{nk} \\log y_{nk}$\n",
    "\n",
    "で計算されており、正解ラベルのみ損失関数として考慮していました。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}