{"notes":[{"id":"lightweight/starnet","category":"lightweight","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"StarNet","subtitle":"一顆星號","author":"Z. Yuan","paperDate":"2024-03-29","publishedOn":"2026-02-04","world":"external/starnet","paperTitle":"Rewrite the Stars","paperUrl":"https://arxiv.org/abs/2403.19967","bibliography":{"title":"Rewrite the Stars","authors":["Xu Ma","Xiyang Dai","Yue Bai","Yizhou Wang","Yun Fu"],"url":"https://arxiv.org/abs/2403.19967","pdfUrl":"https://arxiv.org/pdf/2403.19967v1","codeUrl":"https://github.com/ma-xu/Rewrite-the-Stars","preprint":{"id":"2403.19967","submittedOn":"2024-03-29","url":"https://arxiv.org/abs/2403.19967"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Ma_Rewrite_the_Stars_CVPR_2024_paper.html"},"citation":"@misc{lightweightstarnet2024,\n  title = {{Rewrite the Stars}},\n  author = {Xu Ma and Xiyang Dai and Yue Bai and Yizhou Wang and Yun Fu},\n  year = {2024},\n  eprint = {2403.19967},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2403.19967}\n}\n","citationPath":"/content/papers/citations/lightweight/starnet.11711ddcb0cd.bib"},"thumbnail":{"src":"/content/papers/assets/57e0e49bab986c607a82db36bbc0a45a49e42732719480f1afed603ca4c3e48a.png","width":1011,"height":588},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/starnet.2b3efd3880f1.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"加和乘差多少","text":"加和乘差多少","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"把乘法展開","text":"把乘法展開","depth":3},{"id":"拿掉啟動函數也行","text":"拿掉啟動函數也行","depth":3},{"id":"starnet","text":"StarNet","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"乘法放在哪裡","text":"乘法放在哪裡","depth":3},{"id":"速度的代價","text":"速度的代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"referenced","topic":""},{"id":"lightweight/fasternet","relation":"referenced","topic":""}]},{"id":"lightweight/mobilenet-v2","category":"lightweight","legacyPath":"lightweight/1801-mobilenet-v2","legacySlug":"1801-mobilenet-v2","legacyPaths":[],"title":"MobileNet-V2","subtitle":"瓶頸內的淬煉","author":"Z. Yuan","paperDate":"2018-01-13","publishedOn":"2024-06-27","paperTitle":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","paperUrl":"https://arxiv.org/abs/1801.04381","bibliography":{"title":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","authors":["Mark Sandler","Andrew Howard","Menglong Zhu","Andrey Zhmoginov","Liang-Chieh Chen"],"url":"https://arxiv.org/abs/1801.04381","pdfUrl":"https://arxiv.org/pdf/1801.04381","codeUrl":null,"preprint":{"id":"1801.04381","submittedOn":"2018-01-13","url":"https://arxiv.org/abs/1801.04381"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1801.04381"},"citation":"@misc{lightweightmobilenetv22018,\n  title = {{MobileNetV2: Inverted Residuals and Linear Bottlenecks}},\n  author = {Mark Sandler and Andrew Howard and Menglong Zhu and Andrey Zhmoginov and Liang-Chieh Chen},\n  year = {2018},\n  eprint = {1801.04381},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.04381}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v2.a882707cbaa2.bib"},"thumbnail":{"src":"/content/papers/assets/e6bbfb346bb449f83897d76b69c3711d2e9937d40320a7aab0f3843653a688c0.jpg","width":2332,"height":1324},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/mobilenet-v2.a7a89cbedac8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"線性瓶頸","text":"線性瓶頸","depth":3},{"id":"殘差網路","text":"殘差網路","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"反轉殘差","text":"反轉殘差","depth":3},{"id":"擴張瓶頸論文附錄-a","text":"擴張瓶頸（論文附錄 A）","depth":3},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"圖像分類效能","text":"圖像分類效能","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v1","relation":"background","topic":"行動裝置上的高效率卷積"},{"id":"lightweight/mobilenet-v3","relation":"next","topic":"行動裝置上的高效率卷積"}]},{"id":"lightweight/fasternet","category":"lightweight","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FasterNet","subtitle":"FLOPs 會騙人","author":"Z. Yuan","paperDate":"2023-03-07","publishedOn":"2025-10-23","world":"external/fasternet","paperTitle":"Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks","paperUrl":"https://arxiv.org/abs/2303.03667","bibliography":{"title":"Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks","authors":["Jierun Chen","Shiu-hong Kao","Hao He","Weipeng Zhuo","Song Wen","Chul-Ho Lee","S.-H. Gary Chan"],"url":"https://arxiv.org/abs/2303.03667","pdfUrl":"https://arxiv.org/pdf/2303.03667v3","codeUrl":null,"preprint":{"id":"2303.03667","submittedOn":"2023-03-07","url":"https://arxiv.org/abs/2303.03667"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Chen_Run_Dont_Walk_Chasing_Higher_FLOPS_for_Faster_Neural_Networks_CVPR_2023_paper.html"},"citation":"@misc{lightweightfasternet2023,\n  title = {{Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks}},\n  author = {Jierun Chen and Shiu-hong Kao and Hao He and Weipeng Zhuo and Song Wen and Chul-Ho Lee and S.-H. Gary Chan},\n  year = {2023},\n  eprint = {2303.03667},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2303.03667}\n}\n","citationPath":"/content/papers/citations/lightweight/fasternet.7a313d25fdbe.bib"},"thumbnail":{"src":"/content/papers/assets/f7b24b9388f5bb090f5420a96d640e9f851fd40becb24e6bb9b31ba1ebee4b71.png","width":1012,"height":500},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/fasternet.c8ec7f1bd55f.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"延遲等於-flops-除以-flops","text":"延遲等於 FLOPs 除以 FLOPS","depth":3},{"id":"depthwise-卷積慢在哪裡","text":"depthwise 卷積慢在哪裡","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"部分卷積-pconv","text":"部分卷積 PConv","depth":3},{"id":"再接一個-11-卷積","text":"再接一個 1×1 卷積","depth":3},{"id":"fasternet","text":"FasterNet","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"偵測與分割","text":"偵測與分割","depth":3},{"id":"消融","text":"消融","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"referenced","topic":""},{"id":"lightweight/shufflenet","relation":"referenced","topic":""},{"id":"lightweight/ghostnet","relation":"referenced","topic":""}]}],"text":"這幾年的新架構裡，常常藏著一個逐元素相乘。\n\nTransformer 的 FFN 換成 GLU，兩條分支相乘；Mamba 的區塊裡有一條閘門分支；視覺這邊的 FocalNet、HorNet、VAN 也都在乘。乘法的符號 `*` 長得像星星，這篇論文就把它叫做「星號運算」（star operation）。\n\n每篇論文都說乘法有用，理由卻各說各話。FocalNet 說它像調變或閘門，HorNet 說它帶來高階特徵，VAN 說它是一種卷積式注意力。這篇來自美國東北大學（Northeastern University）和微軟，作者想把這件事說清楚。兩條線性分支乘起來，到底多了什麼？\n\n## 定義問題\n\n### 加和乘差多少\n\n作者先做了一個很單純的區塊。輸入過 LayerNorm 和 depthwise 卷積，分成兩條全連接分支，其中一條接 GELU，兩條再用加法或乘法合起來，最後接一個全連接層和殘差。\n\n[![原論文 Figure 1：左邊是示範用的區塊，由上到下是 Layer Norm、DW-Conv，接著分成兩個 FC 分支，左邊那條帶 GELU，兩條在紅色問號處合併，問號旁標註 sum or star，再接一個 FC 和殘差相加。右邊是長條圖，橫軸是網路寬度 96 到 448，淺藍色長條是 sum 的 ImageNet top-1，淺橘色是 star，紅色三角形虛線是兩者差距，從寬度 96 的約 6.5 個百分點一路降到 448 的約 2.5。](/content/papers/assets/57e0e49bab986c607a82db36bbc0a45a49e42732719480f1afed603ca4c3e48a.png)](/content/papers/assets/57e0e49bab986c607a82db36bbc0a45a49e42732719480f1afed603ca4c3e48a.png)\n\n*原論文 Figure 1（Ma 等人），取自 [**PDF 第 1 頁**](https://arxiv.org/pdf/2403.19967v1#page=1)。圖說未收錄。*\n\n把這個區塊疊 12 層，叫做 DemoNet，只改合併的方式，在 ImageNet 上比較：\n\n| 網路寬度 | 96 | 128 | 160 | 192 | 224 | 256 | 288 |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| 相加 | 51.1% | 58.5% | 62.7% | 66.2% | 69.6% | 71.4% | 72.5% |\n| 相乘 | **57.6%** | **64.0%** | **68.2%** | **71.7%** | **73.9%** | **75.3%** | **76.1%** |\n\n*表：摘自原論文 Table 2，深度固定 12 層。*\n\n每一種寬度都是乘法贏，而且網路越窄贏越多，寬度 96 時差了 6.5 個百分點，到 288 剩 3.6。作者也試了固定寬度、改變深度，差距一直維持在 4 到 6.5 個百分點，沒有隨深度縮小。\n\n乘法在窄網路上特別有用，這點後面會用到。\n\n## 解決問題\n\n### 把乘法展開\n\n先看一個輸出通道。輸入 $x$ 有 $d$ 個通道，最後補一個 1，讓偏差項也能併進權重。兩條分支相乘就是\n\n$$\n(w_1^\\top x)(w_2^\\top x)=\\Big(\\sum_{i=1}^{d+1} w_1^i x_i\\Big)\\Big(\\sum_{j=1}^{d+1} w_2^j x_j\\Big)=\\sum_{i=1}^{d+1}\\sum_{j=1}^{d+1} w_1^i w_2^j\\, x_i x_j .\n$$\n\n展開之後是所有 $x_i x_j$ 的線性組合。$x_i x_j$ 和 $x_j x_i$ 是同一項，合併後共有 $(d+2)(d+1)/2$ 項，大約是 $d^2/2$。換句話說，計算是在 $d$ 維做的，結果卻像是在一個約 $d^2/2$ 維的空間裡做線性運算，而且每一維都是輸入的二次項。\n\n我們用 $d=4$ 驗證一次：\n\n```python\nimport numpy as np\n\nrng = np.random.default_rng(0)\nd = 4\nx = np.append(rng.normal(size=d), 1.0)        # 最後補一個 1，偏差項就能併進權重\nw1, w2 = rng.normal(size=d + 1), rng.normal(size=d + 1)\n\nstar = (w1 @ x) * (w2 @ x)                    # 兩個線性分支逐元素相乘\n\n# 展開成所有 x_i x_j 的線性組合\nA = np.outer(w1, w2)\nterms = {(i, j): (A[i, i] if i == j else A[i, j] + A[j, i]) for i in range(d + 1) for j in range(i, d + 1)}\nexpanded = sum(a * x[i] * x[j] for (i, j), a in terms.items())\n\nprint(f\"直接相乘：{star:.6f}\")\nprint(f\"展開成二次項：{expanded:.6f}\")\nprint(\"輸入維度\", d, \"，展開後的項數\", len(terms))\nsym = (A + A.T) / 2                           # 這 15 個係數其實來自一個對稱矩陣\nprint(\"係數矩陣的秩：\", np.linalg.matrix_rank(sym))\n```\n\n```text\n直接相乘：-0.059305\n展開成二次項：-0.059305\n輸入維度 4 ，展開後的項數 15\n係數矩陣的秩： 2\n```\n\n4 個輸入通道，展開成 15 個二次項。寬度 128 的話是 8385 項。\n\n這和機器學習裡的多項式核很像。多項式核 $k(x_1,x_2)=(\\gamma\\, x_1\\cdot x_2+c)^d$ 不用真的把特徵展開，就能在高維的多項式特徵空間裡做計算。星號運算也是在低維計算，換到高維的表示。\n\n疊更多層，維度會繼續翻倍成長。第二層再把第一層的二次項乘一次，就變成四次項。作者算過，寬度 128、疊 10 層的網路，隱含的維度大約是 $90^{1024}$，可以當成無限大。\n\n不過最後一行輸出值得停一下。15 個係數看起來很多，可是它們全部來自 $w_1$ 和 $w_2$ 這兩個向量，組成的係數矩陣秩只有 2。一個輸出通道沒辦法任意挑選要哪些二次項，維度雖然高，係數卻被綁得很緊。作者在討論裡也提到，隱含維度上的係數分布沒辦法像一般網路的權重那樣自由調整，這可能是「極高的隱含維度只換來中等提升」的原因。\n\n[![原論文 Figure 2：四張二維散點圖，橘色和綠色兩群點排成兩個交錯的月牙形。上排是神經網路，左邊 sum operation 的分界是一條斜直線，右邊 star operation 的分界是彎曲的，比較貼合月牙。下排是 SVM，左邊 Gaussian Kernel 的綠色區域是一團圓弧，右邊 Polynomial Kernel 的分界形狀和上排的 star operation 很像。](/content/papers/assets/b55872c7135a393362fc7046bb8e6019d674440f9d2e201f4eae032c743d3b02.png)](/content/papers/assets/b55872c7135a393362fc7046bb8e6019d674440f9d2e201f4eae032c743d3b02.png)\n\n*原論文 Figure 2（Ma 等人），取自 [**PDF 第 5 頁**](https://arxiv.org/pdf/2403.19967v1#page=5)。圖說未收錄。*\n\n作者在雙月牙資料上畫了決策邊界。相加的網路畫出一條直線，相乘的網路畫出曲線，形狀和多項式核的 SVM 很接近，和高斯核的 SVM 反而不像。\n\n### 拿掉啟動函數也行\n\n如果乘法本身就帶來非線性，那網路裡完全不放啟動函數會怎樣？我們用一個沒有任何啟動函數的小網路試試 XOR，第一、三象限是一類，第二、四象限是另一類，任何一條直線都切不開：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx = torch.rand(2000, 2) * 2 - 1                # 平面上的點，座標在 -1 到 1 之間\ny = (x[:, 0] * x[:, 1] > 0).float()           # 第一、三象限是 1，第二、四象限是 0（XOR）\n\ndef train(combine, width=4):\n    W1, W2 = torch.nn.Linear(2, width), torch.nn.Linear(2, width)\n    head = torch.nn.Linear(width, 1)\n    params = [*W1.parameters(), *W2.parameters(), *head.parameters()]\n    opt = torch.optim.Adam(params, lr=0.05)\n    for _ in range(1000):\n        logit = head(combine(W1(x), W2(x))).squeeze(1)\n        loss = torch.nn.functional.binary_cross_entropy_with_logits(logit, y)\n        opt.zero_grad()\n        loss.backward()\n        opt.step()\n    return ((logit > 0).float() == y).float().mean().item()\n\nprint(f\"兩個分支相加：準確率 {train(lambda a, b: a + b):.1%}\")\nprint(f\"兩個分支相乘：準確率 {train(lambda a, b: a * b):.1%}\")\n```\n\n```text\n兩個分支相加：準確率 53.6%\n兩個分支相乘：準確率 100.0%\n```\n\n相加的版本從頭到尾都是線性函數，只能猜個一半；相乘的版本裡有 $x_1x_2$ 這一項，直接解決。\n\n論文在 DemoNet 上做了同樣的事，寬度 192、深度 12。所有啟動函數拿掉後，相加的網路從 66.2% 掉到 32.4%，相乘的網路從 71.7% 只掉到 70.5%。\n\n### StarNet\n\n既然乘法在窄網路上效果最好，作者就拿它做輕量網路，叫 StarNet。\n\n[![原論文 Figure 3：上方是 StarNet 的整體流程，一張星空圖片輸入後經過 Conv（BN、ReLU6），再依序是 stage1 到 stage4，每個 stage 是一個 Conv 接一疊 Star Blocks，通道數從 d 到 8d、解析度從 H/4 到 H/32，最後接 GAP 和 FC。左下角放大一個 star block：DW-Conv、BN 之後分成兩個 FC 分支，把通道從 d 放大到 4d，其中一條帶 ReLU6，兩條在星號處相乘，再接 FC 縮回 d、BN、DW-Conv，外面一條殘差。右下角圖例寫著 Conv 的 ks=3、stride=2，DW-Conv 的 ks=7、stride=1，星號是逐元素相乘。](/content/papers/assets/55ab4184389c408d6a5688d8bb59a97d8b20245cb9681ca5344bd295116806f6.png)](/content/papers/assets/55ab4184389c408d6a5688d8bb59a97d8b20245cb9681ca5344bd295116806f6.png)\n\n*原論文 Figure 3（Ma 等人），取自 [**PDF 第 6 頁**](https://arxiv.org/pdf/2403.19967v1#page=6)。圖說未收錄。*\n\n架構刻意做得很樸素。四個階段，每個階段開頭用步長 2 的 3×3 卷積降採樣、通道加倍。區塊就是剛才的 DemoNet 區塊再改幾處。LayerNorm 換成 BatchNorm，放在 depthwise 卷積後面，推論時可以併進卷積；GELU 換成 MobileNetV2 用的 ReLU6；區塊尾端再多一個 depthwise 卷積；兩條分支都把通道放大到 4 倍。\n\n重參數化、注意力、SE 這些常見的加分技巧一個都沒有用，作者說是怕蓋過星號運算本身的貢獻。四個尺寸只改第一階段的通道數和每階段的區塊數，最小的 S1 有 2.9M 參數、425M FLOPs，最大的 S4 有 7.5M、1075M。\n\n\n> [!TIP]\n> \n> ReLU6 和倒置殘差可以看 MobileNetV2；同樣主打「實際速度」的輕量網路，可以看 FasterNet：\n> \n> - **[[18.01] MobileNet-V2：瓶頸內的淬煉](/blog/papers/lightweight/mobilenet-v2)**\n> - **[[23.03] FasterNet：FLOPs 會騙人](/blog/papers/lightweight/fasternet)**\n\n\n## 討論\n\n### ImageNet\n\n訓練照 DeiT 的配方，300 個 epoch。延遲在 iPhone 13、P100 GPU 和 Intel E5-2680 CPU 三種裝置上用批次 1 量。\n\n| 模型 | top-1 | 參數量 | FLOPs | iPhone 13 | GPU | CPU |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| MobileOne-S0 | 71.4% | 2.1M | 275M | 0.7 ms | 1.1 ms | 2.2 ms |\n| FasterNet-T0 | 71.9% | 3.9M | 338M | 0.7 ms | 2.5 ms | 5.7 ms |\n| StarNet-S1 | **73.5%** | 2.9M | 425M | 0.7 ms | 2.3 ms | 4.3 ms |\n| MobileOne-S1 | 75.9% | 4.8M | 825M | 0.9 ms | 1.5 ms | 6.0 ms |\n| EfficientNet-B0 | 77.1% | 5.3M | 390M | 1.6 ms | 3.4 ms | 8.8 ms |\n| StarNet-S3 | **77.3%** | 5.8M | 757M | 0.9 ms | 2.7 ms | 6.7 ms |\n| MobileOne-S2 | 77.4% | 7.8M | 1299M | 1.0 ms | 2.0 ms | 8.9 ms |\n| EdgeViT-XS | 77.5% | 6.8M | 1166M | 3.5 ms | 12.1 ms | 18.3 ms |\n| StarNet-S4 | **78.4%** | 7.5M | 1075M | 1.0 ms | 3.7 ms | 9.4 ms |\n\n*表：摘自原論文 Table 6 的部分列。*\n\n在 iPhone 13 上，StarNet 確實漂亮。同樣 0.7 ms，S1 比 MobileOne-S0 高 2.1 個百分點；S4 比 EdgeViT-XS 高 0.9，速度是它的 3.5 倍。\n\n換到 GPU 和 CPU，情況就不太一樣。StarNet-S4 在 P100 上要 3.7 ms，MobileOne-S2 只要 2.0 ms；CPU 上 9.4 對 8.9，也是 MobileOne 快一點。StarNet 的強項在手機；和 EdgeViT 這類混合架構比，三種裝置上都快不少，和 MobileOne 比就要看裝置。\n\n### 乘法放在哪裡\n\n作者把 StarNet-S4 裡的乘法一段一段換回加法：\n\n| 第 1 階段 | 第 2 階段 | 第 3 階段 | 第 4 階段 | top-1 |\n| --- | --- | --- | --- | ---: |\n| 加 | 加 | 加 | 加 | 75.3% |\n| 乘 | 乘 | 加 | 加 | 75.2% |\n| 乘 | 乘 | 乘 | 加 | 76.8% |\n| 加 | 加 | 乘 | 乘 | **78.4%** |\n| 乘 | 乘 | 乘 | 乘 | **78.4%** |\n\n*表：摘自原論文 Table 7 的部分列。*\n\n全部換成加法掉了 3.1 個百分點，可是前兩個階段用不用乘法幾乎沒差，只要後兩個階段用乘法，就和全部用乘法一樣。作者的解釋是前兩個階段太窄，ReLU6 把不少特徵切成 0，乘起來的二次項也跟著變成 0，高維的好處發揮不出來。\n\n啟動函數的位置也試了一輪。完全不放啟動函數是 75.6%，只在一條分支放是 78.4%，兩條都放是 78.0%，乘完再放是 77.0%。只動一條分支最好。\n\n如果只轉換一條分支，另一條直接用輸入 $x$，寫成 $(W_2\\,\\mathrm{act}(W_1 x)) * x$，計算量調到一樣，準確率掉到 74.4%。兩條分支都要經過線性轉換，展開後的二次項才夠多。\n\n### 速度的代價\n\n乘法理論上比加法貴一點。實際量起來，iPhone 和 GPU 上兩者一樣快；CPU 上乘法慢了一些，StarNet-S4 是 9.4 對 8.4 ms。\n\n還有一個論文沒處理的問題。GitHub 上[有使用者問到量化](https://github.com/ma-xu/Rewrite-the-Stars/issues/10)，作者回覆還沒試過；另一位使用者留言提醒，兩個特徵相乘，數值範圍容易超出 INT8 能表示的範圍。部署到手機之前，這一步要自己驗證。\n\n## 結論\n\n這篇把一個大家都在用、卻各有說法的運算，改寫成一行看得懂的展開式。兩條線性分支相乘，等於在約 $d^2/2$ 個二次項上做線性組合，層數一疊，隱含維度就一路往上翻。這也解釋了為什麼乘法在窄網路上贏最多，窄網路最缺的就是維度。\n\n那個秩只有 2 的係數矩陣也說明了上限在哪裡。展開的項很多，能自由調的係數卻很少，「接近無限維」在 DemoNet 和 StarNet 上換來的是 3 到 6.5 個百分點。\n\ntimm 裡就有 `starnet_s1` 到 `starnet_s4` 的預訓練權重。手上的輕量模型不夠準的話，不妨先找找區塊裡哪個加號，改成乘號試試看。\n","imageSizes":{"/content/papers/assets/57e0e49bab986c607a82db36bbc0a45a49e42732719480f1afed603ca4c3e48a.png":{"width":1011,"height":588},"/content/papers/assets/b55872c7135a393362fc7046bb8e6019d674440f9d2e201f4eae032c743d3b02.png":{"width":978,"height":609},"/content/papers/assets/55ab4184389c408d6a5688d8bb59a97d8b20245cb9681ca5344bd295116806f6.png":{"width":1011,"height":693}}}