“`”<\/p>\n
<a class=""url"" href=""https:\/\/www.zhihu.com\/question\/41252833"" target=""_blank"">https:\/\/www.zhihu.com\/question\/41252833<\/a>
\n\u8868\u793a2\u4e2a\u51fd\u6570\u6216\u6982\u7387\u5206\u5e03\u7684\u5dee\u5f02\u6027\uff1a\u5dee\u5f02\u8d8a\u5927\u5219\u76f8\u5bf9\u71b5\u8d8a\u5927\uff0c\u5dee\u5f02\u8d8a\u5c0f\u5219\u76f8\u5bf9\u71b5\u8d8a\u5c0f\uff0c\u7279\u522b\u5730\uff0c\u82e52\u8005\u76f8\u540c\u5219\u71b5\u4e3a0\u3002<\/p>\n
\u4ea4\u53c9\u71b5\u53ef\u5728\u795e\u7ecf\u7f51\u7edc(\u673a\u5668\u5b66\u4e60)\u4e2d\u4f5c\u4e3a\u635f\u5931\u51fd\u6570\uff0cp\u8868\u793a\u771f\u5b9e\u6807\u8bb0\u7684\u5206\u5e03\uff0cq\u5219\u4e3a\u8bad\u7ec3\u540e\u7684\u6a21\u578b\u7684\u9884\u6d4b\u6807\u8bb0\u5206\u5e03\uff0c\u4ea4\u53c9\u71b5\u635f\u5931\u51fd\u6570\u53ef\u4ee5\u8861\u91cfp\u4e0eq\u7684\u76f8\u4f3c\u6027\u3002\u4ea4\u53c9\u71b5\u4f5c\u4e3a\u635f\u5931\u51fd\u6570\u8fd8\u6709\u4e00\u4e2a\u597d\u5904\u662f\u4f7f\u7528sigmoid\u51fd\u6570\u5728\u68af\u5ea6\u4e0b\u964d\u65f6\u80fd\u907f\u514d\u5747\u65b9\u8bef\u5dee\u635f\u5931\u51fd\u6570\u5b66\u4e60\u901f\u7387\u964d\u4f4e\u7684\u95ee\u9898\uff0c\u56e0\u4e3a\u5b66\u4e60\u901f\u7387\u53ef\u4ee5\u88ab\u8f93\u51fa\u7684\u8bef\u5dee\u6240\u63a7\u5236\u3002<\/p>\n
D(p||q) = H(p,q) – H\u00a7 =<\/p>\n
\u300a\u6570\u5b66\u4e4b\u7f8e\u300bP60-68 \u7684\u71b5\uff1a<\/p>\n
<blockquote>\u4efb\u610f\u4e00\u4e2a\u968f\u673a\u53d8\u91cfX\u7684\u71b5\uff08\u4fe1\u606f\u91cf\uff09\uff1aH(X)=-\u7d2f\u52a0P(x)logP(x)
\n\u77e5\u9053\u7684\u4fe1\u606f\u8d8a\u591a\uff0c\u4e0d\u786e\u5b9a\u6027 \u5c31\u8d8a\u5c0f\uff0c\u90a3\u4e48\u8fd9\u4e9b\u76f8\u5173\u7684\u4fe1\u606f\u600e\u4e48\u8868\u793a\u5462\uff1a\u6761\u4ef6\u71b5(Conditional Entropy, CE)\u3002
\nX,Y\u662f\u4e24\u4e2a\u968f\u673a\u53d8\u91cf\uff0c\u5b9a\u4e49\u5728Y\u7684\u6761\u4ef6\u4e0bX\u7684\u6761\u4ef6\u71b5\u4e3a\uff1aH(X|Y)=-\u7d2f\u52a0P(x,y)logP(x|y)\u3002\u6709H(X)>=H(X|Y)
\n\u4fe1\u606f\u7684\u4f5c\u7528\u5728\u4e8e\u6d88\u9664\u4e0d\u786e\u5b9a\u6027\u3002\u627e\u76f8\u5173\u4fe1\u606f\u3002
\n\u90a3\u4e48\u8fd9\u4e9b\u76f8\u5173\u7684\u4fe1\u606f\u5bf9\u4e8e\u51cf\u5c0f\u4e0d\u786e\u5b9a\u6027\u6709\u591a\u5c11\u5e2e\u52a9\u5462\uff0c\u51cf\u5c0f\u4e86\u591a\u5c11\u4e0d\u786e\u5b9a\u6027\u7684\u8861\u91cf\u6807\u51c6\u4e3a\uff1a\u4e92\u4fe1\u606f I(X;Y)=H(X)-H(X|Y)
\n\u53d6\u503c\u4e3a1\uff0c\u5b8c\u5168\u76f8\u5173\uff1b\u53d6\u503c\u4e3a0\uff0c\u5b8c\u5168\u65e0\u5173\u3002
\n\u4ea4\u53c9\u71b5cross_entropy\/\u76f8\u5bf9\u71b5\/KL\u6563\u5ea6\uff1a\u8861\u91cf\u4e24\u4e2a\u53d6\u503c\u4e3a\u6b63\u6570\u7684\u51fd\u6570\u7684\u76f8\u4f3c\u6027
\nKL(f(x)||g(x)) = -\u7d2f\u52a0f(x)log(f(x)\/g(x))
\n\u5b8c\u5168\u76f8\u540c\u7684\u51fd\u6570\uff0c\u4ea4\u53c9\u71b5\u4e3a0\uff1b\u5dee\u5f02\u8d8a\u5927\u4ea4\u53c9\u71b5\u8d8a\u5927
\n<\/blockquote><\/p>\n
<pre><code> "“`
\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
“`” <a class=""url"" […]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[101],"tags":[],"class_list":["post-45143","post","type-post","status-publish","format-standard","hentry","category-c"],"_links":{"self":[{"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/posts\/45143","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/comments?post=45143"}],"version-history":[{"count":1,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/posts\/45143\/revisions"}],"predecessor-version":[{"id":45145,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/posts\/45143\/revisions\/45145"}],"wp:attachment":[{"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/media?parent=45143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/categories?post=45143"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/wx.kaifamiao.info\/index.php\/wp-json\/wp\/v2\/tags?post=45143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}