A framework for parallel and distributed training of neural networks
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenari...
Ausführliche Beschreibung
Gespeichert in:
| Autor*in: |
Scardapane, Simone [verfasserIn] |
|---|
| Format: |
E-Artikel |
|---|---|
| Sprache: |
Englisch |
| Erschienen: |
2017transfer abstract |
|---|
| Schlagwörter: |
|---|
| Umfang: |
13 |
|---|
| Übergeordnetes Werk: |
Enthalten in: Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing - 2012, the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society, Amsterdam |
|---|---|
| Übergeordnetes Werk: |
volume:91 ; year:2017 ; pages:42-54 ; extent:13 |
| Links: |
|---|
| DOI / URN: |
10.1016/j.neunet.2017.04.004 |
|---|
| Katalog-ID: |
ELV035994169 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | ELV035994169 | ||
| 003 | DE-627 | ||
| 005 | 20230625210303.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 180603s2017 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.neunet.2017.04.004 |2 doi | |
| 028 | 5 | 2 | |a GBVA2017014000027.pica |
| 035 | |a (DE-627)ELV035994169 | ||
| 035 | |a (ELSEVIER)S0893-6080(17)30084-9 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 082 | 0 | |a 004 | |
| 082 | 0 | 4 | |a 004 |q DE-600 |
| 082 | 0 | 4 | |a 620 |q VZ |
| 082 | 0 | 4 | |a 610 |q VZ |
| 084 | |a 77.50 |2 bkl | ||
| 100 | 1 | |a Scardapane, Simone |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a A framework for parallel and distributed training of neural networks |
| 264 | 1 | |c 2017transfer abstract | |
| 300 | |a 13 | ||
| 336 | |a nicht spezifiziert |b zzz |2 rdacontent | ||
| 337 | |a nicht spezifiziert |b z |2 rdamedia | ||
| 338 | |a nicht spezifiziert |b zu |2 rdacarrier | ||
| 520 | |a The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. | ||
| 520 | |a The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. | ||
| 650 | 7 | |a Networks |2 Elsevier | |
| 650 | 7 | |a Parallel computing |2 Elsevier | |
| 650 | 7 | |a Distributed learning |2 Elsevier | |
| 650 | 7 | |a Neural network |2 Elsevier | |
| 700 | 1 | |a Di Lorenzo, Paolo |4 oth | |
| 773 | 0 | 8 | |i Enthalten in |n Elsevier |t Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |d 2012 |d the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society |g Amsterdam |w (DE-627)ELV016218965 |
| 773 | 1 | 8 | |g volume:91 |g year:2017 |g pages:42-54 |g extent:13 |
| 856 | 4 | 0 | |u https://doi.org/10.1016/j.neunet.2017.04.004 |3 Volltext |
| 912 | |a GBV_USEFLAG_U | ||
| 912 | |a GBV_ELV | ||
| 912 | |a SYSFLAG_U | ||
| 912 | |a SSG-OLC-PHA | ||
| 936 | b | k | |a 77.50 |j Psychophysiologie |q VZ |
| 951 | |a AR | ||
| 952 | |d 91 |j 2017 |h 42-54 |g 13 | ||
| 953 | |2 045F |a 004 | ||
| author_variant |
s s ss |
|---|---|
| matchkey_str |
scardapanesimonedilorenzopaolo:2017----:faeokoprleaditiuetann |
| hierarchy_sort_str |
2017transfer abstract |
| bklnumber |
77.50 |
| publishDate |
2017 |
| allfields |
10.1016/j.neunet.2017.04.004 doi GBVA2017014000027.pica (DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 DE-627 ger DE-627 rakwb eng 004 004 DE-600 620 VZ 610 VZ 77.50 bkl Scardapane, Simone verfasserin aut A framework for parallel and distributed training of neural networks 2017transfer abstract 13 nicht spezifiziert zzz rdacontent nicht spezifiziert z rdamedia nicht spezifiziert zu rdacarrier The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier Di Lorenzo, Paolo oth Enthalten in Elsevier Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing 2012 the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society Amsterdam (DE-627)ELV016218965 volume:91 year:2017 pages:42-54 extent:13 https://doi.org/10.1016/j.neunet.2017.04.004 Volltext GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA 77.50 Psychophysiologie VZ AR 91 2017 42-54 13 045F 004 |
| spelling |
10.1016/j.neunet.2017.04.004 doi GBVA2017014000027.pica (DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 DE-627 ger DE-627 rakwb eng 004 004 DE-600 620 VZ 610 VZ 77.50 bkl Scardapane, Simone verfasserin aut A framework for parallel and distributed training of neural networks 2017transfer abstract 13 nicht spezifiziert zzz rdacontent nicht spezifiziert z rdamedia nicht spezifiziert zu rdacarrier The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier Di Lorenzo, Paolo oth Enthalten in Elsevier Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing 2012 the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society Amsterdam (DE-627)ELV016218965 volume:91 year:2017 pages:42-54 extent:13 https://doi.org/10.1016/j.neunet.2017.04.004 Volltext GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA 77.50 Psychophysiologie VZ AR 91 2017 42-54 13 045F 004 |
| allfields_unstemmed |
10.1016/j.neunet.2017.04.004 doi GBVA2017014000027.pica (DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 DE-627 ger DE-627 rakwb eng 004 004 DE-600 620 VZ 610 VZ 77.50 bkl Scardapane, Simone verfasserin aut A framework for parallel and distributed training of neural networks 2017transfer abstract 13 nicht spezifiziert zzz rdacontent nicht spezifiziert z rdamedia nicht spezifiziert zu rdacarrier The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier Di Lorenzo, Paolo oth Enthalten in Elsevier Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing 2012 the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society Amsterdam (DE-627)ELV016218965 volume:91 year:2017 pages:42-54 extent:13 https://doi.org/10.1016/j.neunet.2017.04.004 Volltext GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA 77.50 Psychophysiologie VZ AR 91 2017 42-54 13 045F 004 |
| allfieldsGer |
10.1016/j.neunet.2017.04.004 doi GBVA2017014000027.pica (DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 DE-627 ger DE-627 rakwb eng 004 004 DE-600 620 VZ 610 VZ 77.50 bkl Scardapane, Simone verfasserin aut A framework for parallel and distributed training of neural networks 2017transfer abstract 13 nicht spezifiziert zzz rdacontent nicht spezifiziert z rdamedia nicht spezifiziert zu rdacarrier The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier Di Lorenzo, Paolo oth Enthalten in Elsevier Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing 2012 the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society Amsterdam (DE-627)ELV016218965 volume:91 year:2017 pages:42-54 extent:13 https://doi.org/10.1016/j.neunet.2017.04.004 Volltext GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA 77.50 Psychophysiologie VZ AR 91 2017 42-54 13 045F 004 |
| allfieldsSound |
10.1016/j.neunet.2017.04.004 doi GBVA2017014000027.pica (DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 DE-627 ger DE-627 rakwb eng 004 004 DE-600 620 VZ 610 VZ 77.50 bkl Scardapane, Simone verfasserin aut A framework for parallel and distributed training of neural networks 2017transfer abstract 13 nicht spezifiziert zzz rdacontent nicht spezifiziert z rdamedia nicht spezifiziert zu rdacarrier The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier Di Lorenzo, Paolo oth Enthalten in Elsevier Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing 2012 the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society Amsterdam (DE-627)ELV016218965 volume:91 year:2017 pages:42-54 extent:13 https://doi.org/10.1016/j.neunet.2017.04.004 Volltext GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA 77.50 Psychophysiologie VZ AR 91 2017 42-54 13 045F 004 |
| language |
English |
| source |
Enthalten in Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing Amsterdam volume:91 year:2017 pages:42-54 extent:13 |
| sourceStr |
Enthalten in Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing Amsterdam volume:91 year:2017 pages:42-54 extent:13 |
| format_phy_str_mv |
Article |
| bklname |
Psychophysiologie |
| institution |
findex.gbv.de |
| topic_facet |
Networks Parallel computing Distributed learning Neural network |
| dewey-raw |
004 |
| isfreeaccess_bool |
false |
| container_title |
Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |
| authorswithroles_txt_mv |
Scardapane, Simone @@aut@@ Di Lorenzo, Paolo @@oth@@ |
| publishDateDaySort_date |
2017-01-01T00:00:00Z |
| hierarchy_top_id |
ELV016218965 |
| dewey-sort |
14 |
| id |
ELV035994169 |
| language_de |
englisch |
| fullrecord |
<?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>01000caa a22002652 4500</leader><controlfield tag="001">ELV035994169</controlfield><controlfield tag="003">DE-627</controlfield><controlfield tag="005">20230625210303.0</controlfield><controlfield tag="007">cr uuu---uuuuu</controlfield><controlfield tag="008">180603s2017 xx |||||o 00| ||eng c</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1016/j.neunet.2017.04.004</subfield><subfield code="2">doi</subfield></datafield><datafield tag="028" ind1="5" ind2="2"><subfield code="a">GBVA2017014000027.pica</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627)ELV035994169</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(ELSEVIER)S0893-6080(17)30084-9</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-627</subfield><subfield code="b">ger</subfield><subfield code="c">DE-627</subfield><subfield code="e">rakwb</subfield></datafield><datafield tag="041" ind1=" " ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="082" ind1="0" ind2=" "><subfield code="a">004</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">004</subfield><subfield code="q">DE-600</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">620</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">610</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">77.50</subfield><subfield code="2">bkl</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Scardapane, Simone</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">A framework for parallel and distributed training of neural networks</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="c">2017transfer abstract</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">13</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">zzz</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">z</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">zu</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Networks</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Parallel computing</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Distributed learning</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Neural network</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Di Lorenzo, Paolo</subfield><subfield code="4">oth</subfield></datafield><datafield tag="773" ind1="0" ind2="8"><subfield code="i">Enthalten in</subfield><subfield code="n">Elsevier</subfield><subfield code="t">Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing</subfield><subfield code="d">2012</subfield><subfield code="d">the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society</subfield><subfield code="g">Amsterdam</subfield><subfield code="w">(DE-627)ELV016218965</subfield></datafield><datafield tag="773" ind1="1" ind2="8"><subfield code="g">volume:91</subfield><subfield code="g">year:2017</subfield><subfield code="g">pages:42-54</subfield><subfield code="g">extent:13</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doi.org/10.1016/j.neunet.2017.04.004</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_USEFLAG_U</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ELV</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SYSFLAG_U</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SSG-OLC-PHA</subfield></datafield><datafield tag="936" ind1="b" ind2="k"><subfield code="a">77.50</subfield><subfield code="j">Psychophysiologie</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="951" ind1=" " ind2=" "><subfield code="a">AR</subfield></datafield><datafield tag="952" ind1=" " ind2=" "><subfield code="d">91</subfield><subfield code="j">2017</subfield><subfield code="h">42-54</subfield><subfield code="g">13</subfield></datafield><datafield tag="953" ind1=" " ind2=" "><subfield code="2">045F</subfield><subfield code="a">004</subfield></datafield></record></collection>
|
| author |
Scardapane, Simone |
| spellingShingle |
Scardapane, Simone ddc 004 ddc 620 ddc 610 bkl 77.50 Elsevier Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network A framework for parallel and distributed training of neural networks |
| authorStr |
Scardapane, Simone |
| ppnlink_with_tag_str_mv |
@@773@@(DE-627)ELV016218965 |
| format |
electronic Article |
| dewey-ones |
004 - Data processing & computer science 620 - Engineering & allied operations 610 - Medicine & health |
| delete_txt_mv |
keep |
| author_role |
aut |
| collection |
elsevier |
| remote_str |
true |
| illustrated |
Not Illustrated |
| topic_title |
004 004 DE-600 620 VZ 610 VZ 77.50 bkl A framework for parallel and distributed training of neural networks Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network Elsevier |
| topic |
ddc 004 ddc 620 ddc 610 bkl 77.50 Elsevier Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network |
| topic_unstemmed |
ddc 004 ddc 620 ddc 610 bkl 77.50 Elsevier Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network |
| topic_browse |
ddc 004 ddc 620 ddc 610 bkl 77.50 Elsevier Networks Elsevier Parallel computing Elsevier Distributed learning Elsevier Neural network |
| format_facet |
Elektronische Aufsätze Aufsätze Elektronische Ressource |
| format_main_str_mv |
Text Zeitschrift/Artikel |
| carriertype_str_mv |
zu |
| author2_variant |
p d pd |
| hierarchy_parent_title |
Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |
| hierarchy_parent_id |
ELV016218965 |
| dewey-tens |
000 - Computer science, knowledge & systems 620 - Engineering 610 - Medicine & health |
| hierarchy_top_title |
Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |
| isfreeaccess_txt |
false |
| familylinks_str_mv |
(DE-627)ELV016218965 |
| title |
A framework for parallel and distributed training of neural networks |
| ctrlnum |
(DE-627)ELV035994169 (ELSEVIER)S0893-6080(17)30084-9 |
| title_full |
A framework for parallel and distributed training of neural networks |
| author_sort |
Scardapane, Simone |
| journal |
Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |
| journalStr |
Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing |
| lang_code |
eng |
| isOA_bool |
false |
| dewey-hundreds |
000 - Computer science, information & general works 600 - Technology |
| recordtype |
marc |
| publishDateSort |
2017 |
| contenttype_str_mv |
zzz |
| container_start_page |
42 |
| author_browse |
Scardapane, Simone |
| container_volume |
91 |
| physical |
13 |
| class |
004 004 DE-600 620 VZ 610 VZ 77.50 bkl |
| format_se |
Elektronische Aufsätze |
| author-letter |
Scardapane, Simone |
| doi_str_mv |
10.1016/j.neunet.2017.04.004 |
| dewey-full |
004 620 610 |
| title_sort |
a framework for parallel and distributed training of neural networks |
| title_auth |
A framework for parallel and distributed training of neural networks |
| abstract |
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. |
| abstractGer |
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. |
| abstract_unstemmed |
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach. |
| collection_details |
GBV_USEFLAG_U GBV_ELV SYSFLAG_U SSG-OLC-PHA |
| title_short |
A framework for parallel and distributed training of neural networks |
| url |
https://doi.org/10.1016/j.neunet.2017.04.004 |
| remote_bool |
true |
| author2 |
Di Lorenzo, Paolo |
| author2Str |
Di Lorenzo, Paolo |
| ppnlink |
ELV016218965 |
| mediatype_str_mv |
z |
| isOA_txt |
false |
| hochschulschrift_bool |
false |
| author2_role |
oth |
| doi_str |
10.1016/j.neunet.2017.04.004 |
| up_date |
2024-07-06T19:00:59.406Z |
| _version_ |
1803857385916203008 |
| fullrecord_marcxml |
<?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>01000caa a22002652 4500</leader><controlfield tag="001">ELV035994169</controlfield><controlfield tag="003">DE-627</controlfield><controlfield tag="005">20230625210303.0</controlfield><controlfield tag="007">cr uuu---uuuuu</controlfield><controlfield tag="008">180603s2017 xx |||||o 00| ||eng c</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1016/j.neunet.2017.04.004</subfield><subfield code="2">doi</subfield></datafield><datafield tag="028" ind1="5" ind2="2"><subfield code="a">GBVA2017014000027.pica</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627)ELV035994169</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(ELSEVIER)S0893-6080(17)30084-9</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-627</subfield><subfield code="b">ger</subfield><subfield code="c">DE-627</subfield><subfield code="e">rakwb</subfield></datafield><datafield tag="041" ind1=" " ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="082" ind1="0" ind2=" "><subfield code="a">004</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">004</subfield><subfield code="q">DE-600</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">620</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="082" ind1="0" ind2="4"><subfield code="a">610</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">77.50</subfield><subfield code="2">bkl</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Scardapane, Simone</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">A framework for parallel and distributed training of neural networks</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="c">2017transfer abstract</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">13</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">zzz</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">z</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="a">nicht spezifiziert</subfield><subfield code="b">zu</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification–decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., ℓ 2 norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Networks</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Parallel computing</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Distributed learning</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="650" ind1=" " ind2="7"><subfield code="a">Neural network</subfield><subfield code="2">Elsevier</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Di Lorenzo, Paolo</subfield><subfield code="4">oth</subfield></datafield><datafield tag="773" ind1="0" ind2="8"><subfield code="i">Enthalten in</subfield><subfield code="n">Elsevier</subfield><subfield code="t">Regulatory design for RES-E support mechanisms: Learning curves, market structure, and burden-sharing</subfield><subfield code="d">2012</subfield><subfield code="d">the official journal of the International Neural Network Society, European Neural Network Society and Japanese Neural Network Society</subfield><subfield code="g">Amsterdam</subfield><subfield code="w">(DE-627)ELV016218965</subfield></datafield><datafield tag="773" ind1="1" ind2="8"><subfield code="g">volume:91</subfield><subfield code="g">year:2017</subfield><subfield code="g">pages:42-54</subfield><subfield code="g">extent:13</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doi.org/10.1016/j.neunet.2017.04.004</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_USEFLAG_U</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ELV</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SYSFLAG_U</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SSG-OLC-PHA</subfield></datafield><datafield tag="936" ind1="b" ind2="k"><subfield code="a">77.50</subfield><subfield code="j">Psychophysiologie</subfield><subfield code="q">VZ</subfield></datafield><datafield tag="951" ind1=" " ind2=" "><subfield code="a">AR</subfield></datafield><datafield tag="952" ind1=" " ind2=" "><subfield code="d">91</subfield><subfield code="j">2017</subfield><subfield code="h">42-54</subfield><subfield code="g">13</subfield></datafield><datafield tag="953" ind1=" " ind2=" "><subfield code="2">045F</subfield><subfield code="a">004</subfield></datafield></record></collection>
|
| score |
7.40049 |