|
14 | 14 | }, |
15 | 15 | { |
16 | 16 | "cell_type": "code", |
17 | | - "execution_count": 6, |
| 17 | + "execution_count": 1, |
18 | 18 | "metadata": { |
19 | 19 | "collapsed": true, |
20 | 20 | "deletable": true, |
|
582 | 582 | }, |
583 | 583 | { |
584 | 584 | "cell_type": "markdown", |
585 | | - "metadata": {}, |
| 585 | + "metadata": { |
| 586 | + "deletable": true, |
| 587 | + "editable": true |
| 588 | + }, |
586 | 589 | "source": [ |
587 | 590 | "## Distance Functions\n", |
588 | 591 | "\n", |
|
597 | 600 | "cell_type": "code", |
598 | 601 | "execution_count": 2, |
599 | 602 | "metadata": { |
600 | | - "collapsed": false |
| 603 | + "collapsed": false, |
| 604 | + "deletable": true, |
| 605 | + "editable": true |
601 | 606 | }, |
602 | 607 | "outputs": [ |
603 | 608 | { |
|
619 | 624 | }, |
620 | 625 | { |
621 | 626 | "cell_type": "markdown", |
622 | | - "metadata": {}, |
| 627 | + "metadata": { |
| 628 | + "deletable": true, |
| 629 | + "editable": true |
| 630 | + }, |
623 | 631 | "source": [ |
624 | 632 | "### Euclidean Distance (`euclidean_distance`)\n", |
625 | 633 | "\n", |
|
630 | 638 | "cell_type": "code", |
631 | 639 | "execution_count": 13, |
632 | 640 | "metadata": { |
633 | | - "collapsed": false |
| 641 | + "collapsed": false, |
| 642 | + "deletable": true, |
| 643 | + "editable": true |
634 | 644 | }, |
635 | 645 | "outputs": [ |
636 | 646 | { |
|
652 | 662 | }, |
653 | 663 | { |
654 | 664 | "cell_type": "markdown", |
655 | | - "metadata": {}, |
| 665 | + "metadata": { |
| 666 | + "deletable": true, |
| 667 | + "editable": true |
| 668 | + }, |
656 | 669 | "source": [ |
657 | 670 | "### Hamming Distance (`hamming_distance`)\n", |
658 | 671 | "\n", |
|
663 | 676 | "cell_type": "code", |
664 | 677 | "execution_count": 4, |
665 | 678 | "metadata": { |
666 | | - "collapsed": false |
| 679 | + "collapsed": false, |
| 680 | + "deletable": true, |
| 681 | + "editable": true |
667 | 682 | }, |
668 | 683 | "outputs": [ |
669 | 684 | { |
|
685 | 700 | }, |
686 | 701 | { |
687 | 702 | "cell_type": "markdown", |
688 | | - "metadata": {}, |
| 703 | + "metadata": { |
| 704 | + "deletable": true, |
| 705 | + "editable": true |
| 706 | + }, |
689 | 707 | "source": [ |
690 | 708 | "### Mean Boolean Error (`mean_boolean_error`)\n", |
691 | 709 | "\n", |
|
696 | 714 | "cell_type": "code", |
697 | 715 | "execution_count": 9, |
698 | 716 | "metadata": { |
699 | | - "collapsed": false |
| 717 | + "collapsed": false, |
| 718 | + "deletable": true, |
| 719 | + "editable": true |
700 | 720 | }, |
701 | 721 | "outputs": [ |
702 | 722 | { |
|
718 | 738 | }, |
719 | 739 | { |
720 | 740 | "cell_type": "markdown", |
721 | | - "metadata": {}, |
| 741 | + "metadata": { |
| 742 | + "deletable": true, |
| 743 | + "editable": true |
| 744 | + }, |
722 | 745 | "source": [ |
723 | 746 | "### Mean Error (`mean_error`)\n", |
724 | 747 | "\n", |
|
729 | 752 | "cell_type": "code", |
730 | 753 | "execution_count": 10, |
731 | 754 | "metadata": { |
732 | | - "collapsed": false |
| 755 | + "collapsed": false, |
| 756 | + "deletable": true, |
| 757 | + "editable": true |
733 | 758 | }, |
734 | 759 | "outputs": [ |
735 | 760 | { |
|
751 | 776 | }, |
752 | 777 | { |
753 | 778 | "cell_type": "markdown", |
754 | | - "metadata": {}, |
| 779 | + "metadata": { |
| 780 | + "deletable": true, |
| 781 | + "editable": true |
| 782 | + }, |
755 | 783 | "source": [ |
756 | 784 | "### Mean Square Error (`ms_error`)\n", |
757 | 785 | "\n", |
|
762 | 790 | "cell_type": "code", |
763 | 791 | "execution_count": 11, |
764 | 792 | "metadata": { |
765 | | - "collapsed": false |
| 793 | + "collapsed": false, |
| 794 | + "deletable": true, |
| 795 | + "editable": true |
766 | 796 | }, |
767 | 797 | "outputs": [ |
768 | 798 | { |
|
784 | 814 | }, |
785 | 815 | { |
786 | 816 | "cell_type": "markdown", |
787 | | - "metadata": {}, |
| 817 | + "metadata": { |
| 818 | + "deletable": true, |
| 819 | + "editable": true |
| 820 | + }, |
788 | 821 | "source": [ |
789 | 822 | "### Root of Mean Square Error (`rms_error`)\n", |
790 | 823 | "\n", |
|
795 | 828 | "cell_type": "code", |
796 | 829 | "execution_count": 12, |
797 | 830 | "metadata": { |
798 | | - "collapsed": false |
| 831 | + "collapsed": false, |
| 832 | + "deletable": true, |
| 833 | + "editable": true |
799 | 834 | }, |
800 | 835 | "outputs": [ |
801 | 836 | { |
|
1062 | 1097 | "\n", |
1063 | 1098 | "The Perceptron is a linear classifier. It works the same way as a neural network with no hidden layers (just input and output). First it trains its weights given a dataset and then it can classify a new item by running it through the network.\n", |
1064 | 1099 | "\n", |
1065 | | - "You can think of it as a single neuron. It has *n* synapses, each with its own weight. Each synapse corresponds to one item feature. Perceptron multiplies each item feature with the corresponding synapse weight and then adds them together (aka, the dot product) and checks whether this value is greater than the threshold. If yes, it returns 1. It returns 0 otherwise.\n", |
| 1100 | + "Its input layer consists of the the item features, while the output layer consists of nodes (also called neurons). Each node in the output layer has *n* synapses (for every item feature), each with its own weight. Then, the nodes find the dot product of the item features and the synapse weights. These values then pass through an activation function (usually a sigmoid). Finally, we pick the largest of the values and we return its index.\n", |
| 1101 | + "\n", |
| 1102 | + "Note that in classification problems each node represents a class. The final classification is the class/node with the max output value.\n", |
1066 | 1103 | "\n", |
| 1104 | + "Below you can see a single node/neuron in the outer layer. With *f* we denote the item features, with *w* the synapse weights, then inside the node we have the dot product and the activation function, *g*." |
| 1105 | + ] |
| 1106 | + }, |
| 1107 | + { |
| 1108 | + "cell_type": "markdown", |
| 1109 | + "metadata": {}, |
| 1110 | + "source": [ |
1067 | 1111 | "" |
1068 | 1112 | ] |
1069 | 1113 | }, |
|
1076 | 1120 | "source": [ |
1077 | 1121 | "### Implementation\n", |
1078 | 1122 | "\n", |
1079 | | - "First, we train (calculate) the weights given a dataset, using the `BackPropagationLearner` function of `learning.py`. We then return a function, `predict`, which we will use in the future to classify a new item. The function computes the (algebraic) dot product of the item with the calculated weights. If the result is greater than a predefined threshold (usually 0.5, 0 or 1), it returns 1. If it is less than the threshold, it returns 0.\n", |
1080 | | - "\n", |
1081 | | - "NOTE: The current implementation of the algorithm classifies an item into one of two classes. It is a binary classifier and will not work well for multi-class datasets." |
| 1123 | + "First, we train (calculate) the weights given a dataset, using the `BackPropagationLearner` function of `learning.py`. We then return a function, `predict`, which we will use in the future to classify a new item. The function computes the (algebraic) dot product of the item with the calculated weights for each node in the outer layer. Then it picks the greatest value and classifies the item in the corresponding class." |
1082 | 1124 | ] |
1083 | 1125 | }, |
1084 | 1126 | { |
1085 | 1127 | "cell_type": "code", |
1086 | | - "execution_count": 6, |
| 1128 | + "execution_count": 2, |
1087 | 1129 | "metadata": { |
1088 | 1130 | "collapsed": true, |
1089 | 1131 | "deletable": true, |
1090 | 1132 | "editable": true |
1091 | 1133 | }, |
1092 | 1134 | "outputs": [], |
1093 | 1135 | "source": [ |
1094 | | - "def PerceptronLearner(dataset, learning_rate=0.01, epochs=100):\n", |
1095 | | - " \"\"\"Logistic Regression, NO hidden layer\"\"\"\n", |
1096 | | - " i_units = len(dataset.inputs)\n", |
1097 | | - " o_units = 1 # As of now, dataset.target gives only one index.\n", |
1098 | | - " hidden_layer_sizes = []\n", |
1099 | | - " raw_net = network(i_units, hidden_layer_sizes, o_units)\n", |
1100 | | - " learned_net = BackPropagationLearner(dataset, raw_net, learning_rate, epochs)\n", |
1101 | | - "\n", |
1102 | | - " def predict(example):\n", |
1103 | | - " # Input nodes\n", |
1104 | | - " i_nodes = learned_net[0]\n", |
1105 | | - "\n", |
1106 | | - " # Activate input layer\n", |
1107 | | - " for v, n in zip(example, i_nodes):\n", |
1108 | | - " n.value = v\n", |
1109 | | - "\n", |
1110 | | - " # Forward pass\n", |
1111 | | - " for layer in learned_net[1:]:\n", |
1112 | | - " for node in layer:\n", |
1113 | | - " inc = [n.value for n in node.inputs]\n", |
1114 | | - " in_val = dotproduct(inc, node.weights)\n", |
1115 | | - " node.value = node.activation(in_val)\n", |
1116 | | - "\n", |
1117 | | - " # Hypothesis\n", |
1118 | | - " o_nodes = learned_net[-1]\n", |
1119 | | - " pred = [o_nodes[i].value for i in range(o_units)]\n", |
1120 | | - " return 1 if pred[0] >= 0.5 else 0\n", |
1121 | | - "\n", |
1122 | | - " return predict" |
| 1136 | + "%psource PerceptronLearner" |
1123 | 1137 | ] |
1124 | 1138 | }, |
1125 | 1139 | { |
|
1129 | 1143 | "editable": true |
1130 | 1144 | }, |
1131 | 1145 | "source": [ |
1132 | | - "The weights are trained from the `BackPropagationLearner`. Note that the perceptron is a one-layer neural network, without any hidden layers. So, in `BackPropagationLearner`, we will pass no hidden layers. From that function we get our network, which is just one node, with the weights calculated.\n", |
1133 | | - "\n", |
1134 | | - "`PerceptronLearner` returns `predict`, a function that can be used to classify a new item.\n", |
| 1146 | + "Note that the Perceptron is a one-layer neural network, without any hidden layers. So, in `BackPropagationLearner`, we will pass no hidden layers. From that function we get our network, which is just one layer, with the weights calculated.\n", |
1135 | 1147 | "\n", |
1136 | | - "That function passes the input/example through the network, calculating the dot product of the input and the weights. If that value is greater than or equal to 0.5, it returns 1. Otherwise it returns 0." |
| 1148 | + "That function `predict` passes the input/example through the network, calculating the dot product of the input and the weights for each node and returns the class with the max dot product." |
1137 | 1149 | ] |
1138 | 1150 | }, |
1139 | 1151 | { |
|
1145 | 1157 | "source": [ |
1146 | 1158 | "### Example\n", |
1147 | 1159 | "\n", |
1148 | | - "We will train the Perceptron on the iris dataset. Because, though, the algorithm is a binary classifier (which means it classifies an item in one of two classes) and the iris dataset has three classes, we need to transform the dataset into a proper form, with only two classes. Therefore, we will remove the third and final class of the dataset, *Virginica*.\n", |
1149 | | - "\n", |
1150 | | - "Then, we will try and classify the item/flower with measurements of 5,3,1,0.1." |
| 1160 | + "We will train the Perceptron on the iris dataset. Because though the `BackPropagationLearner` works with integer indexes and not strings, we need to convert class names to integers. Then, we will try and classify the item/flower with measurements of 5, 3, 1, 0.1." |
1151 | 1161 | ] |
1152 | 1162 | }, |
1153 | 1163 | { |
1154 | 1164 | "cell_type": "code", |
1155 | | - "execution_count": 7, |
| 1165 | + "execution_count": 10, |
1156 | 1166 | "metadata": { |
1157 | 1167 | "collapsed": false, |
1158 | 1168 | "deletable": true, |
|
1169 | 1179 | ], |
1170 | 1180 | "source": [ |
1171 | 1181 | "iris = DataSet(name=\"iris\")\n", |
1172 | | - "iris.remove_examples(\"virginica\")\n", |
1173 | 1182 | "iris.classes_to_numbers()\n", |
1174 | 1183 | "\n", |
1175 | 1184 | "perceptron = PerceptronLearner(iris)\n", |
1176 | | - "print(perceptron([5,3,1,0.1]))" |
| 1185 | + "print(perceptron([5, 3, 1, 0.1]))" |
1177 | 1186 | ] |
1178 | 1187 | }, |
1179 | 1188 | { |
|
1183 | 1192 | "editable": true |
1184 | 1193 | }, |
1185 | 1194 | "source": [ |
1186 | | - "The output is 0, which means the item is classified in the first class, *setosa*. This is indeed correct. Note that the Perceptron algorithm is not perfect and may produce false classifications." |
| 1195 | + "The output is 0, which means the item is classified in the first class, \"Setosa\". This is indeed correct. Note that the Perceptron algorithm is not perfect and may produce false classifications." |
1187 | 1196 | ] |
1188 | 1197 | }, |
1189 | 1198 | { |
|
0 commit comments