{"notes":[{"id":"contrastive-learning/examplar-cnn","category":"contrastive-learning","legacyPath":"contrastive-learning/1406-examplar-cnn","legacySlug":"1406-examplar-cnn","legacyPaths":[],"title":"Exemplar CNN","subtitle":"尋找不變的特徵","author":"Z. Yuan","paperDate":"2014-06-26","publishedOn":"2025-01-22","paperTitle":"Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks","paperUrl":"https://arxiv.org/abs/1406.6909","bibliography":{"title":"Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks","authors":["Alexey Dosovitskiy","Philipp Fischer","Jost Tobias Springenberg","Martin Riedmiller","Thomas Brox"],"url":"https://arxiv.org/abs/1406.6909","pdfUrl":"https://arxiv.org/pdf/1406.6909","codeUrl":null,"preprint":{"id":"1406.6909","submittedOn":"2014-06-26","url":"https://arxiv.org/abs/1406.6909"},"publication":{"venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","shortVenue":"IEEE TPAMI","year":2016,"kind":"journal","url":"https://doi.org/10.1109/tpami.2015.2496141"},"citation":"@misc{contrastivelearningexamplarcnn2014,\n  title = {{Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks}},\n  author = {Alexey Dosovitskiy and Philipp Fischer and Jost Tobias Springenberg and Martin Riedmiller and Thomas Brox},\n  year = {2014},\n  eprint = {1406.6909},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1406.6909}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/examplar-cnn.9f293122852b.bib"},"thumbnail":{"src":"/content/papers/assets/52a4c5931d81eeedf2a0857944f10ddc3749d7195e81184efa2a80bc736d062d.jpg","width":1080,"height":418},"locale":"zh-hant","path":"/content/papers/zh-hant/contrastive-learning/examplar-cnn.306f0384165e.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"實驗配置","text":"實驗配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"和其他方法的比較","text":"和其他方法的比較","depth":3},{"id":"消融實驗-代理類別數量","text":"消融實驗-代理類別數量","depth":3},{"id":"消融實驗-樣本數量","text":"消融實驗-樣本數量","depth":3},{"id":"消融實驗-影像增強類型","text":"消融實驗-影像增強類型","depth":3},{"id":"消融實驗-參數配置","text":"消融實驗-參數配置","depth":3},{"id":"濾波器可視化","text":"濾波器可視化","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/invaspread","relation":"comparison","topic":"如何定義自監督預測任務"},{"id":"contrastive-learning/cpc","relation":"comparison","topic":"如何定義自監督預測任務"}]},{"id":"contrastive-learning/invaspread","category":"contrastive-learning","legacyPath":"contrastive-learning/1904-invaspread","legacySlug":"1904-invaspread","legacyPaths":[],"title":"InvaSpread","subtitle":"難題下的蛛絲馬跡","author":"Z. Yuan","paperDate":"2019-04-06","publishedOn":"2025-01-23","paperTitle":"Unsupervised Embedding Learning via Invariant and Spreading Instance Feature","paperUrl":"https://arxiv.org/abs/1904.03436","bibliography":{"title":"Unsupervised Embedding Learning via Invariant and Spreading Instance Feature","authors":["Mang Ye","Xu Zhang","Pong C. Yuen","Shih-Fu Chang"],"url":"https://arxiv.org/abs/1904.03436","pdfUrl":"https://arxiv.org/pdf/1904.03436","codeUrl":null,"preprint":{"id":"1904.03436","submittedOn":"2019-04-06","url":"https://arxiv.org/abs/1904.03436"},"publication":{"venue":"CVPR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1904.03436"},"citation":"@misc{contrastivelearninginvaspread2019,\n  title = {{Unsupervised Embedding Learning via Invariant and Spreading Instance Feature}},\n  author = {Mang Ye and Xu Zhang and Pong C. Yuen and Shih-Fu Chang},\n  year = {2019},\n  eprint = {1904.03436},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1904.03436}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/invaspread.a46b3f57ac16.bib"},"thumbnail":{"src":"/content/papers/assets/135f248ffa5c98e798e9cd5d75ab7cdb38fda8130eae2511547941d70f316165.jpg","width":1596,"height":524},"locale":"zh-hant","path":"/content/papers/zh-hant/contrastive-learning/invaspread.83a304b7fac5.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"損失函數","text":"損失函數","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"和其他方法的比較","text":"和其他方法的比較","depth":3},{"id":"未知類別的實驗","text":"未知類別的實驗","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/examplar-cnn","relation":"comparison","topic":"如何定義自監督預測任務"},{"id":"contrastive-learning/cpc","relation":"comparison","topic":"如何定義自監督預測任務"}]},{"id":"contrastive-learning/cpc","category":"contrastive-learning","legacyPath":"contrastive-learning/1807-cpc","legacySlug":"1807-cpc","legacyPaths":[],"title":"CPC","subtitle":"未知生，焉知死？","author":"Z. Yuan","paperDate":"2018-07-10","publishedOn":"2025-01-16","paperTitle":"Representation Learning with Contrastive Predictive Coding","paperUrl":"https://arxiv.org/abs/1807.03748","bibliography":{"title":"Representation Learning with Contrastive Predictive Coding","authors":["Aaron van den Oord","Yazhe Li","Oriol Vinyals"],"url":"https://arxiv.org/abs/1807.03748","pdfUrl":"https://arxiv.org/pdf/1807.03748","codeUrl":null,"preprint":{"id":"1807.03748","submittedOn":"2018-07-10","url":"https://arxiv.org/abs/1807.03748"},"publication":null,"citation":"@misc{contrastivelearningcpc2018,\n  title = {{Representation Learning with Contrastive Predictive Coding}},\n  author = {Aaron van den Oord and Yazhe Li and Oriol Vinyals},\n  year = {2018},\n  eprint = {1807.03748},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1807.03748}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/cpc.4cc0d1d40232.bib"},"thumbnail":{"src":"/content/papers/assets/a60b1a92f6088b36ddee7c87fed67d1e73769e003c30805f963c6b284d0c3a86.jpg","width":1382,"height":626},"locale":"zh-hant","path":"/content/papers/zh-hant/contrastive-learning/cpc.97fe876188bf.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"infonce","text":"InfoNCE","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"語音辨識領域應用","text":"語音辨識領域應用","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"語者辨識可視化","text":"語者辨識可視化","depth":3},{"id":"影像辨識領域應用","text":"影像辨識領域應用","depth":3},{"id":"影像特徵可視化","text":"影像特徵可視化","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/examplar-cnn","relation":"comparison","topic":"如何定義自監督預測任務"},{"id":"contrastive-learning/invaspread","relation":"comparison","topic":"如何定義自監督預測任務"}]}],"text":"讀論文的時候，發現許多研究都參考了這篇論文的想法，所以我們花點時間來看看它。\n\n這篇論文的概念很簡單，但有許多實驗結果能夠給我們帶來一些想法。\n\n## 定義問題\n\n監督式學習的優點同時也是它的缺點：需要大量的標註資料。而在一些特定的應用場景下，這種資料可能難以取得，或者價格昂貴。\n\n而且如果我們想要在不同的任務之間進行遷移學習（transfer learning），例如從物體分類轉換到特徵描述子匹配（descriptor matching），監督式學習的效果可能不盡理想。\n\n既然不理想，一個簡單的想法就是：或許我們不使用資料標籤呢？\n\n但如果不使用資料標籤的話，模型該怎麼訓練呢？\n\n作者說：我們需要有個代理任務。\n\n## 解決問題\n\n作者在論文中提出了一種完全不需要人工標註的訓練方法，稱作 **Exemplar-CNN**。\n\n核心概念非常簡單：\n\n1. **隨機抽取影像塊**：為了更專注於含有較多物體資訊的區域，會按影像塊內的平均平方梯度大小加權取樣，從未標註影像中擷取若干個 $32 \\times 32$ 大小的影像塊。\n\n\n> [!TIP]\n> \n> 這不是倒傳遞那個「梯度」，而是影像的梯度。作者認為，影像中的梯度可以幫助我們找到更有意義的影像塊。\n\n\n2. **對影像塊進行各種隨機轉換**：例如平移、縮放、旋轉、顏色調整、對比度調整等，得到多個「變形版本」。這些轉換明確定義了網路要學到的不變性（如對旋轉不敏感、對色彩變化不敏感等）。\n3. **形成「代理類別」並進行判別式學習**：每個原始影像塊及其所有變形版本，被視為同一個代理類別中的「正樣本」。\n\n系統在大量影像中重複此過程，產生許多代理類別，並訓練一個 CNN 去判別「哪些圖像塊（含各種變形版本）屬於同一代理類別，哪些屬於其他代理類別」。如此一來，模型既能學到如何分辨不同影像塊，又對「同一個影像塊的各種變形」具備不變性。\n\n在論文中所使用的轉換列表如下：\n\n- **平移**：在垂直或水平方向上，平移距離不超過圖像塊大小的 20%。\n- **縮放**：按比例縮放圖像塊，縮放係數介於 0.7 至 1.4 之間。\n- **旋轉**：旋轉角度在 ±20 度之內。\n- **對比度 1**：每個像素在所有像素主成分方向上的投影，乘以 0.5 至 2 之間的因子（對主成分方向獨立，對圖像塊內像素相同）。\n- **對比度 2**：\n  - 將 HSV 顏色表示的飽和度和明度（S 和 V 分量）提升至 0.25 至 4 之間的冪次。\n  - 再乘以 0.7 至 1.4 之間的因子，並加上 −0.1 到 0.1 之間的偏移值。\n- **顏色**：對 HSV 顏色表示的色調（H 分量），加上 −0.1 到 0.1 之間的偏移值。\n\n\n> [!TIP]\n> \n> 在消融實驗的章節，作者會驗證這些轉換的有效性。\n\n\n經過裁切後的訓練資料大概長這樣：（以 STL-10 為例）\n\n\n\n![dataset](/content/papers/assets/52a4c5931d81eeedf2a0857944f10ddc3749d7195e81184efa2a80bc736d062d.jpg)\n\n\n\n### 實驗配置\n\n作者為了比較 **Exemplar-CNN** 與先前的無監督特徵學習方法，基於以下數據集進行了一系列實驗：STL-10、CIFAR-10、Caltech-101 和 Caltech-256。\n\n除了資料集轉移實驗，代理類別的訓練資料取自 STL-10 的未標註影像；評估分類時，再用各數據集的標註資料訓練線性 SVM。\n\n數據集的數據量、類別數量、圖像尺寸等細節如下：\n\n- **分類數量**：\n  - STL-10 和 CIFAR-10：10 類。\n  - Caltech-101：101 類。\n  - Caltech-256：256 類。\n- **圖像尺寸處理**：\n  - CIFAR-10：從 $32 \\times 32$ 調整至 $64 \\times 64$。\n  - Caltech-101：調整為 $150 \\times 150$。\n  - Caltech-256：調整為 $256 \\times 256$。\n\n接著作者做了幾個網路架構，如下設計：\n\n- **小型網路（64c5-64c5-128f）**：\n  - 兩層卷積層（各 64 個濾波器）。\n  - 一層全連接層（128 單元）。\n  - 最後接 Softmax 層。\n- **中型網路（64c5-128c5-256c5-512f）**：\n  - 三層卷積層（濾波器數量分別為 64, 128, 256）。\n  - 一層全連接層（512 單元）。\n- **大型網路（92c5-256c5-512c5-1024f）**：\n  - 三層卷積層（濾波器數量分別為 92, 256, 512）。\n  - 一層全連接層（1024 單元）。\n- **卷積與池化細節**：\n  - 所有卷積濾波器對應輸入的 $5 \\times 5$ 區域。\n  - $2 \\times 2$ 最大池化應用於前兩層卷積層。\n- **其他技術**：\n  - 全連接層使用 Dropout。\n  - 基於 Caffe 實現，訓練細節見附錄 B.2。\n\n測試方法是將網路應用於任意尺寸圖像，計算特徵映射，並根據數據集選用適當的池化方法：\n\n- STL-10 和 CIFAR-10：四象限最大池化（每個特徵映射得 4 個值）。\n- Caltech-101 和 Caltech-256：三層空間金字塔池化（$1+4+16=21$值）。\n\n最後使用一對多線性支持向量機（SVM）進行分類。\n\n## 討論\n\n### 和其他方法的比較\n\n\n\n![result](/content/papers/assets/8beeea477c413bdd28ee69403a980d8c4d22550dd6f0024b27e06d4ed7d8249b.jpg)\n\n\n\n上表比較了 **Exemplar-CNN** 與其他無監督特徵學習方法在不同數據集上的性能。\n\n實驗結果顯示，較大的 **Exemplar-CNN** 在表中各數據集的無監督特徵學習方法比較裡均取得最佳結果；其中 STL-10 的 74.2% 也高於表列監督式方法的 70.1%！其他數據集的監督式結果仍較高，兩類方法的訓練條件也不同。\n\n### 消融實驗-代理類別數量\n\n\n\n![num_classes](/content/papers/assets/7160448ec9abe144de27e04fd75a28fc27b1d49b319e3392af59a0beb687ae0c.jpg)\n\n\n\n作者首先測試了代理類別數量對性能的影響。\n\n實驗設計將代理類別數量 $N$ 從 50 調整至 32000，並觀察了分類準確率的變化。\n\n結果如上圖，可以看到 STL-10 分類準確率隨代理類別數量增加而提升，直到 4000～8000 類左右達到最佳後穩定或下降；同時代理分類的驗證錯誤率持續升高。作者認為類別數量過多時，相似或重複的樣本更難區分，可能降低特徵的分類效果。\n\n\n> [!TIP]\n> \n> 這個實驗沒有比較特徵歸一化的影響，因此不能單憑此圖判定曲線停止上升的原因。\n\n\n### 消融實驗-樣本數量\n\n\n\n![num_samples](/content/papers/assets/2225aee7d0b3dbc4053b3de583132afea8ec0690f17eea71b8df96bb6671eafe.jpg)\n\n\n\n再來是每類代理樣本數量 $K$ 的影響。\n\n作者調整 $K$ 在 1 到 300 之間，觀察分類準確率的變化。\n\n結果顯示，STL-10 準確率隨每類樣本數量增加而提升，在 100 樣本附近趨於穩定。作者認為此時已足以近似論文式 (3) 的期望目標，再增加樣本數對這組實驗的影響不大。\n\n### 消融實驗-影像增強類型\n\n\n\n![augmentation](/content/papers/assets/d5d7ac3dc0915a68389f73b704ba8ccaa310e27e36dc931d1bfb10d0071b9738.jpg)\n\n\n\n作者進一步探討了不同影像增強類型對性能的影響。\n\n實驗設計固定原始「種子」樣本，調整應用於代理數據的轉換類型，包括縮放、旋轉、平移、顏色變化、對比度變化等。\n\n圖中的負值是移除該類轉換後，相對於使用全部轉換的準確率差。移除平移、顏色變化或對比度變化時，下降通常較大；縮放與旋轉的影響較小。各數據集對轉換的敏感程度並不相同。\n\n### 消融實驗-參數配置\n\n\n\n![result](/content/papers/assets/ad5a378e1fe37c38f5f5768e23ad6086d329151f76fa94dbbd04655bc8ecae21.jpg)\n\n\n\n上表展示了 **Exemplar-CNN** 使用不同配置參數，在不同數據集上的性能。\n\n表中部分加寬或加深的配置提高了分類準確率，但並非網路越大就越好：四個卷積層的 `64c5-64c5-64c5-64c5-128f` 比三層的 `64c5-64c5-64c5-128f` 差。同一架構從 8000 增至 16000 個代理類別時，各測試欄的變化也不一致。\n\n### 濾波器可視化\n\n\n\n![visualization](/content/papers/assets/73db85a33fb1cfd89576321307a73088b60e67d178c1557090005c4a8655be12.jpg)\n\n\n\n上圖展示了 `64c5-64c5-128f` 網路第一層在不同數據集上的代理數據進行訓練時所學到的濾波器，排列方式如下：\n\n- **頂部**：來自 STL-10 數據集的代理數據。\n- **中間**：來自 CIFAR-10 數據集的代理數據。\n- **底部**：來自 Caltech-101 數據集的代理數據。\n\n分析一下上圖所呈現的資訊，可以看到不同訓練數據集得到的第一層濾波器，在色塊與局部紋理上有所不同；單憑這張圖還不能判定濾波器學到了哪些物體語義。\n\n作者另以分類實驗比較資料來源，發現用各數據集自己的影像塊訓練通常最好，但跨數據集的差距不大；這比單看濾波器更能支持特徵轉移的說法。\n\n## 結論\n\n這篇論文提出了一種是基於判別目標的無監督特徵學習方法，無需依賴物體類別標籤。作者通過數據增強生成代理標籤，並通過影像增強來定義網路需學習的特徵不變性。\n\n實驗結果表明，Exemplar-CNN 所學特徵在分類準確率上取得了顯著提升，並且判別目標相較於以往的無監督學習目標更具優勢。\n\n\n> [!TIP]\n> \n> 比較早期的對比學習研究大多會引用這篇論文，雖然現在看來論文中有許多可以優化的地方，但它無疑給後續的研究帶來非常多的啟發，值得一看。\n","imageSizes":{"/content/papers/assets/52a4c5931d81eeedf2a0857944f10ddc3749d7195e81184efa2a80bc736d062d.jpg":{"width":1080,"height":418},"/content/papers/assets/8beeea477c413bdd28ee69403a980d8c4d22550dd6f0024b27e06d4ed7d8249b.jpg":{"width":1738,"height":440},"/content/papers/assets/7160448ec9abe144de27e04fd75a28fc27b1d49b319e3392af59a0beb687ae0c.jpg":{"width":996,"height":688},"/content/papers/assets/2225aee7d0b3dbc4053b3de583132afea8ec0690f17eea71b8df96bb6671eafe.jpg":{"width":916,"height":664},"/content/papers/assets/d5d7ac3dc0915a68389f73b704ba8ccaa310e27e36dc931d1bfb10d0071b9738.jpg":{"width":1044,"height":596},"/content/papers/assets/ad5a378e1fe37c38f5f5768e23ad6086d329151f76fa94dbbd04655bc8ecae21.jpg":{"width":1536,"height":662},"/content/papers/assets/73db85a33fb1cfd89576321307a73088b60e67d178c1557090005c4a8655be12.jpg":{"width":876,"height":708}}}
