-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
338 lines (278 loc) · 18.1 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-27141560-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-27141560-1');
</script>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="Website of Heyang Qin, Senior Researcher at Microsoft">
<meta name="author" content="">
<title>Heyang Qin - Microsoft</title>
<!-- Bootstrap core CSS -->
<link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<!-- Custom fonts for this template -->
<link href="https://fonts.googleapis.com/css?family=Saira+Extra+Condensed:100,200,300,400,500,600,700,800,900" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600,600i,700,700i,800,800i" rel="stylesheet">
<link href="vendor/font-awesome/css/font-awesome.min.css" rel="stylesheet">
<link href="vendor/devicons/css/devicons.min.css" rel="stylesheet">
<link href="vendor/simple-line-icons/css/simple-line-icons.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="css/resume.min.css" rel="stylesheet">
</head>
<body id="page-top">
<nav class="navbar navbar-expand-lg navbar-dark bg-primary fixed-top" id="sideNav">
<a class="navbar-brand js-scroll-trigger" href="#page-top">
<span class="d-block d-lg-none">Heyang Qin - Microsoft</span>
<span class="d-none d-lg-block">
<img class="img-fluid img-profile rounded-circle mx-auto mb-2" src="img/profile.jpg" alt="">
</span>
</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav">
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#about">About</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#research">Research</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#publications">Publications</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#experience">Experience</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#education">Education</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#teaching">Teaching</a>
</li>
<!-- <li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#skills">Skills</a>
</li> -->
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#interests">Interests</a>
</li>
<!-- <li class="nav-item"> -->
<!-- <a class="nav-link js-scroll-trigger" href="#awards">Awards</a> -->
<!-- </li> -->
</ul>
</div>
</nav>
<div class="container-fluid p-0 bg">
<section class="resume-section p-3 p-lg-5 d-flex d-column" id="about">
<div class="my-auto">
<h1 class="mb-0">Heyang
<span class="text-primary">Qin</span>
</h1>
<div class="subheading mb-5">Microsoft · Redmond, WA ·
<a href="mailto:[email protected]">[email protected]</a>
·
<a href="mailto:[email protected]">[email protected]</a>
</div>
<!-- <div class="subheading mb-5">Office: Scrugham Engineering and Mines (SEM) 207</div> -->
<p class="mb-5">I work as a Senior Researcher at Microsoft, where I'm a part of the DeepSpeed team. My research focuses on enhancing the efficiency of large-scale machine learning systems, covering both training and inference. I completed my Ph.D. from the University of Nevada, Reno in 2022, under the guidance of Dr. Feng Yan and Dr. Lei Yang. Before that, I earned my Bachelor's degree from the University of Electronic Science and Technology of China in 2017. My work is dedicated to exploring ways to improving the efficiency of large models.</p>
<ul class="list-inline list-social-icons mb-0">
<li class="list-inline-item">
<a href="CV.pdf" target="_blank">
<span class="fa-stack fa-lg">
<img src="img/cv icon.png" style="width:52px;height:52x;border:0;">
</span>
</a>
</li>
<li class="list-inline-item">
<a href="https://scholar.google.com/citations?user=0azIceIAAAAJ" target="_blank">
<span class="fa-stack fa-lg">
<img src="img/google scholar icon 2.png" style="width:56px;height:56x;border:0;">
</span>
</a>
</li>
<li class="list-inline-item">
<a href="https://github.com/HeyangQin" target="_blank">
<span class="fa-stack fa-lg">
<img src="img/github icon.png" style="width:56px;height:56x;border:0;">
</span>
</a>
</li>
<li class="list-inline-item">
<a href="https://www.linkedin.com/in/heyangqin/en">
<span class="fa-stack fa-lg">
<img src="img/linkedin icon.png" style="width:58px;height:58x;border:0;">
</span>
</a>
</li>
</ul>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="research">
<div class="my-auto">
<h2 class="mb-5">Research</h2>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<p><b><font size="4">SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement</font></b><p>
<!-- <div class="subheading mb-3">University of Nevada, Reno</div> -->
<p>Large scale training requires massive parallelism to finish the training within a reasonable amount of time. To support massive parallelism, large batch training is the key enabler but often at the cost of generalization performance.
We propose a fully automated and lightweight adaptive batching methodology to enable fine-grained batch size adaption (e.g., at a mini-batch level) that can achieve state-of-the-art performance with record breaking batch sizes.
The core component of our method is a lightweight yet efficient representation of the critical gradient noise information.
Extensive evaluations on popular benchmarks (e.g., CIFAR10, ImageNet, and BERT-Large) demonstrate that the proposed methodology outperforms state-of-the-art methodologies using adaptive batching approaches or hand-tuned static strategies in both performance and batch size.
Particularly, we achieve a new state-of-the-art batch size of 78k in BERT-Large pretraining with SQuAD score 90.69 compared to 90.58 reported in previous state-of-the-art with 59k batch size.</p>
</div>
<!-- <div class="resume-date text-md-right">
<span class="text-primary">August 2017 - Present</span>
</div> -->
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<p><b><font size="4">Region Based Reinforcement Learning Scheduling Framework for MLaaS</font></b><p>
<!-- <div class="subheading mb-3">University of Nevada, Reno</div> -->
<p>The parallelism settings in Machine Learning as a Service (MLaaS) have critical impact on the system performance. It is a challenge to tune the parallel config because of the complex dependency and large search space. We propose a region based reinforcement learning (RRL) approach that can converge to near-optimal config magnitude faster than the traditional reinforcement learning. The proposed RRL is prototyped and evaluated using several real-world machine learning workloads. Both theoretical analysis and experiment evaluation show that RRL outperforms state-of-the-art tuning algorithms for MLaaS.</p>
</div>
<!-- <div class="resume-date text-md-right">
<span class="text-primary">August 2017 - Present</span>
</div> -->
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<p><b><font size="4">RRL Plus: Adaptive Region Based Reinforcement Learning for Machine Learning</font></b><p>
<!-- <div class="subheading mb-3">University of Nevada, Reno</div> -->
<p>RRL is sensitive to region size. Excessive region size leads to large performance gap between RRL solution and optimal one whereas inadequate region size leads to longer learning process. We further expand the Region Based Reinforcement Learning algorithm by Bayesian optimization and heuristic algorithm and enable it to automatically adjust the region size to achieve fast converge and near-optimal solution.</p>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<p><b><font size="4">Semi-supervised Learning for Large Scale Noisy Data</font></b><p>
<!-- <div class="subheading mb-3">University of Nevada, Reno</div> -->
<p>A large amount of data is available for scientific use. Unfortunately, accurate labels of training data are usually manually-labeled and expensive, leading to insufficient labeled data to train machine learning model. We solve this problem by using generative model to resemble multiple state-of-the-art models to achieve better detection from very noisy training data.</p>
</div>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="publications">
<h2 class="mb-5">Publications</h2>
<p>Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He, <a href="2021_NIPS_SimiGrad.pdf" target="_blank"><u>SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement</u></a>, <i>in Proceedings of the Neural Information Processing Systems 2021 (<b>NeurIPS 2021</b>)</i>, Virtual, December, 2021 (<b>Acceptance rate: 2371/9122=26%</b>). [<a href="NIPS 2021 SimiGrad.pptx" target="_blank"><u>Slides</u></a>]</p>
<p>Heyang Qin, Syed Zawad, Yanqi Zhou, Sanjay Padhi, Lei Yang, and Feng Yan, Reinforcement Learning Empowered MLaaS Scheduling for Serving Intelligent Internet of Things, <i>IEEE Internet of Things Journal</i>, 2020 (<b>Impact factor: 9.515</b>).</p>
<p>Heyang Qin, Syed Zawad, Yanqi Zhou, Lei Yang, Dongfang Zhao, Feng Yan, <a href="QinZZYZ019.pdf" target="_blank"><u>Swift Machine Learning Model Serving Scheduling: A Region Based Reinforcement Learning Approach</u></a>, <i>in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (<b>SC 2019</b>)</i>, Denver, CO, USA, Nov, 2019 (<b>Acceptance rate: 78/344=22%</b>). [<a href="SC19 Presentation Heyang Qin.pptx" target="_blank"><u>Slides</u></a>]</p>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="experience">
<div class="my-auto">
<h2 class="mb-5">Experience</h2>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0">Researcher</h3>
<div class="subheading mb-3">Microsoft</div>
<p>I am currently working as a researcher of the Microsoft DeepSpeed team. My work focus on improving the efficiency and scalability of large scale maching learning systems by means of system optimizations as well as CUDA kernel optimization.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">January 2023 - Present</span>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0">Research Intern</h3>
<div class="subheading mb-3">Microsoft</div>
<p>I worked as a research intern of the Microsoft DeepSpeed team. My work focus on using adaptive methods to optimize large scale machine learning in terms of performance and scalability. My recent work SimiGrad achieves a record-breaking large batch size of 77k in BERT pretrain while keeping sota model performance.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">October 2020 - December 2022</span>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0">Research Assistant</h3>
<div class="subheading mb-3">University of Nevada, Reno</div>
<p>Duing this period, I work on researches about reinforcement learning and deep learning as well as their applications on cloud computing.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">August 2017 - December 2022</span>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0">Teaching Assistant</h3>
<div class="subheading mb-3">University of Nevada, Reno</div>
<p>As a teaching assistant, my job includes teaching lab sections, holding office hours and grading.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">August 2017 - January 2021</span>
</div>
</div>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="education">
<div class="my-auto">
<h2 class="mb-5">Education</h2>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0">University of Nevada, Reno</h3>
<div class="subheading mb-3">Ph.D.</div>
<div>Computer Science and Engineering</div>
<p>GPA: 4.00</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">August 2017 - December 2022</span>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row">
<div class="resume-content mr-auto">
<h3 class="mb-0">University of Electronic Science and Technology of China</h3>
<div class="subheading mb-3">Bachelor</div>
<div>Automation Engineering</div>
<!-- <p>GPA: 3.56</p> -->
</div>
<div class="resume-date text-md-right">
<span class="text-primary">August 2013 - May 2017</span>
</div>
</div>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="teaching">
<div class="my-auto">
<h2 class="mb-5">Teaching</h2>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0"><a href="https://catalog.unr.edu/preview_course_nopop.php?catoid=24&coid=188010" target="_blank">CPE 201 - Digital Design</a></h3>
<div class="subheading mb-3">University of Nevada, Reno</div>
<p>I work as a teaching assistant on CPE 201 with Dr. Hung La and Dr. Siming Liu.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">August 2017 - June 2019</span>
</div>
</div>
<div class="resume-item d-flex flex-column flex-md-row mb-5">
<div class="resume-content mr-auto">
<h3 class="mb-0"><a href="https://catalog.unr.edu/preview_course_nopop.php?catoid=13&coid=99089" target="_blank">ENGR 100 - Introduction to Engineering Design</a></h3>
<div class="subheading mb-3">University of Nevada, Reno</div>
<p>I work as a teaching assistant on ENGR 100 with Dr. Ann-Marie Vollstedt, Prof. Kelly Keselica and Dr. Adam Kirn.</p>
</div>
<div class="resume-date text-md-right">
<span class="text-primary">June 2019 - December 2019</span>
</div>
</div>
</div>
</section>
<section class="resume-section p-3 p-lg-5 d-flex flex-column" id="interests">
<div class="my-auto">
<h2 class="mb-5">Personal Interests</h2>
<p>Apart from being a researcher, I partcipate in multiple open-source projects where I contribute codes, translation, etc. I also play table tennis and volleyball occasionally in my free time.</p>
<p class="mb-0">For indoor time, I usually enjoy my time with video games and detective fictions. I also have strong interest and rich knowledge about archaic Chinese and Chinese ancient literature.</p>
</div>
</section>
</div>
<!-- Bootstrap core JavaScript -->
<script src="vendor/jquery/jquery.min.js"></script>
<script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<!-- Plugin JavaScript -->
<script src="vendor/jquery-easing/jquery.easing.min.js"></script>
<!-- Custom scripts for this template -->
<script src="js/resume.min.js"></script>
</body>
</html>