1
00:00:04,960 --> 00:00:19,999
[Music]

2
00:00:20,320 --> 00:00:23,760
hello

3
00:00:21,279 --> 00:00:26,679
everyone um I'm very excited to be here

4
00:00:23,760 --> 00:00:29,160
at pikon you today I attended this

5
00:00:26,679 --> 00:00:31,759
conference uh three times in the past as

6
00:00:29,160 --> 00:00:34,440
a as a listener and today is my first

7
00:00:31,759 --> 00:00:36,480
time participating as a speaker and this

8
00:00:34,440 --> 00:00:37,440
is also my first time speaking at a

9
00:00:36,480 --> 00:00:40,280
large

10
00:00:37,440 --> 00:00:41,600
conference and um as this is my first

11
00:00:40,280 --> 00:00:43,200
time speaking let me start by

12
00:00:41,600 --> 00:00:46,760
introducing

13
00:00:43,200 --> 00:00:49,399
myself so I'm artam um I'm one of the

14
00:00:46,760 --> 00:00:52,000
software engineerings at um software

15
00:00:49,399 --> 00:00:54,000
engineerings directors at cover genius

16
00:00:52,000 --> 00:00:56,920
however uh one of the stickers on my

17
00:00:54,000 --> 00:00:58,960
laptop says developer and this is how I

18
00:00:56,920 --> 00:01:01,320
prefer to identify myself at least for

19
00:00:58,960 --> 00:01:04,839
the purpose of a Tech

20
00:01:01,320 --> 00:01:07,880
conference um I joined cgus in 2017 when

21
00:01:04,839 --> 00:01:10,119
it was a small startup of uh around 20

22
00:01:07,880 --> 00:01:12,840
people and help it to grow

23
00:01:10,119 --> 00:01:16,840
into uh a global leader in embedded

24
00:01:12,840 --> 00:01:20,320
insurance with uh over 700 employees uh

25
00:01:16,840 --> 00:01:21,880
worldwide today overall I have around 15

26
00:01:20,320 --> 00:01:23,759
years of experience in software

27
00:01:21,880 --> 00:01:25,200
development and most most of it uh

28
00:01:23,759 --> 00:01:28,000
building backend

29
00:01:25,200 --> 00:01:30,079
applications and uh during this time I

30
00:01:28,000 --> 00:01:34,360
had opportunities to lead um a few few

31
00:01:30,079 --> 00:01:38,360
different teams including product QA uh

32
00:01:34,360 --> 00:01:41,960
devops and um infrastructure teams I'm

33
00:01:38,360 --> 00:01:44,159
also one of uh C Jango uh Meetup Group

34
00:01:41,960 --> 00:01:46,840
organizers and as as I have this

35
00:01:44,159 --> 00:01:50,000
opportunity today I want to promote our

36
00:01:46,840 --> 00:01:52,719
uh User Group um if you live in Sydney

37
00:01:50,000 --> 00:01:54,920
or nearby we have our next event on 5th

38
00:01:52,719 --> 00:01:57,960
of December so uh it will be at

39
00:01:54,920 --> 00:02:01,560
Microsoft reactor so please come uh say

40
00:01:57,960 --> 00:02:04,479
hi to me and other Jung engineers in

41
00:02:01,560 --> 00:02:07,280
Sydney uh on personal front I a husband

42
00:02:04,479 --> 00:02:10,160
and a dad of two kids my da is almost

43
00:02:07,280 --> 00:02:14,160
nine and my son is three and a half year

44
00:02:10,160 --> 00:02:17,000
old um and as you can imagine uh work in

45
00:02:14,160 --> 00:02:18,959
family keeps me uh pretty busy but I

46
00:02:17,000 --> 00:02:21,560
also try to find time for my hobby which

47
00:02:18,959 --> 00:02:23,959
is martial arts I have been practicing

48
00:02:21,560 --> 00:02:27,720
Brazilian J Jitsu for many years um and

49
00:02:23,959 --> 00:02:30,480
Judah as well um and I like watching um

50
00:02:27,720 --> 00:02:32,920
UFC and other martial arts competitions

51
00:02:30,480 --> 00:02:36,160
at my spare

52
00:02:32,920 --> 00:02:38,440
time uh my today's talk is based on my

53
00:02:36,160 --> 00:02:41,560
experience working uh in a team that

54
00:02:38,440 --> 00:02:44,159
developed uh excore application at cover

55
00:02:41,560 --> 00:02:45,959
genius um and excore is a central

56
00:02:44,159 --> 00:02:49,080
service of xcover

57
00:02:45,959 --> 00:02:50,840
platform uh it is a Jango andf API that

58
00:02:49,080 --> 00:02:54,120
distributes embedded uh protection

59
00:02:50,840 --> 00:02:56,159
products and cover genus Partners use

60
00:02:54,120 --> 00:02:58,480
this API to integrate those products in

61
00:02:56,159 --> 00:03:00,640
their websites uh mobile applications

62
00:02:58,480 --> 00:03:03,799
and other platforms

63
00:03:00,640 --> 00:03:06,799
um we started excore development in 2018

64
00:03:03,799 --> 00:03:08,640
and the fun fact is that uh only one

65
00:03:06,799 --> 00:03:11,640
engineer out of three in the original

66
00:03:08,640 --> 00:03:13,400
team had previous experience with Jango

67
00:03:11,640 --> 00:03:16,560
uh and it wasn't

68
00:03:13,400 --> 00:03:18,280
me um despite this fact we were able to

69
00:03:16,560 --> 00:03:20,720
launch our first production integration

70
00:03:18,280 --> 00:03:24,200
already next year and then one more in

71
00:03:20,720 --> 00:03:26,000
2020 before pandemic nearly BR C

72
00:03:24,200 --> 00:03:28,879
Virginia's business to

73
00:03:26,000 --> 00:03:30,720
hold it was crazy roller coaster

74
00:03:28,879 --> 00:03:35,239
throughout 202 20 and

75
00:03:30,720 --> 00:03:38,000
2021 um but uh in 2021 we were able to

76
00:03:35,239 --> 00:03:39,040
launch 20 new Integrations and a few new

77
00:03:38,000 --> 00:03:41,439
business

78
00:03:39,040 --> 00:03:43,720
verticals and it was the year when we

79
00:03:41,439 --> 00:03:45,159
also started using multiple databases uh

80
00:03:43,720 --> 00:03:47,879
in

81
00:03:45,159 --> 00:03:50,640
excore migrating from one database to

82
00:03:47,879 --> 00:03:53,159
multiple uh helped us to continue scale

83
00:03:50,640 --> 00:03:57,599
excore and

84
00:03:53,159 --> 00:04:02,239
um um and take take on board even more

85
00:03:57,599 --> 00:04:04,280
Partners um in 2022 cgus doubled the

86
00:04:02,239 --> 00:04:06,799
number of partner uh Partnerships and

87
00:04:04,280 --> 00:04:09,840
the growth continued throughout 2023 and

88
00:04:06,799 --> 00:04:12,159
2024 making next cover one of the global

89
00:04:09,840 --> 00:04:14,680
leading embedded Insurance platforms uh

90
00:04:12,159 --> 00:04:17,479
today with hundreds of Partners and

91
00:04:14,680 --> 00:04:19,519
millions of global

92
00:04:17,479 --> 00:04:21,759
customers a couple of words why we

93
00:04:19,519 --> 00:04:21,759
choose

94
00:04:21,800 --> 00:04:27,600
jungo um the alternative approach uh the

95
00:04:25,400 --> 00:04:29,800
the alternative proposal that that I put

96
00:04:27,600 --> 00:04:31,320
was to use flask and SQL alchy I was was

97
00:04:29,800 --> 00:04:35,639
more familiar with this stack as I use

98
00:04:31,320 --> 00:04:37,440
it in the previous application and um I

99
00:04:35,639 --> 00:04:39,000
guess back then we we barely understood

100
00:04:37,440 --> 00:04:40,520
the complexity of the system that we

101
00:04:39,000 --> 00:04:42,639
were about to start

102
00:04:40,520 --> 00:04:44,479
building uh but one of the things that

103
00:04:42,639 --> 00:04:46,919
we knew for sure if the project is

104
00:04:44,479 --> 00:04:49,199
successful um it will grow FAS and we

105
00:04:46,919 --> 00:04:51,000
will need to scale it so before

106
00:04:49,199 --> 00:04:53,360
committing into using jungo we run a few

107
00:04:51,000 --> 00:04:56,160
performance benchmarks to ensure that

108
00:04:53,360 --> 00:04:59,960
jungo wouldn't be slower than flask plus

109
00:04:56,160 --> 00:05:02,320
scale alchemia and I um still have helps

110
00:04:59,960 --> 00:05:05,360
in those results that received but uh in

111
00:05:02,320 --> 00:05:08,680
our configuration Jango even slightly

112
00:05:05,360 --> 00:05:10,039
outperformed um the other stack uh and

113
00:05:08,680 --> 00:05:13,000
of course we considered many other

114
00:05:10,039 --> 00:05:16,560
factors but one of the uh major

115
00:05:13,000 --> 00:05:21,800
arguments was that um Instagram was

116
00:05:16,560 --> 00:05:23,120
originally built with jungo um so yeah I

117
00:05:21,800 --> 00:05:26,479
don't know how it sounds but in the

118
00:05:23,120 --> 00:05:29,360
startup World um the time is very

119
00:05:26,479 --> 00:05:32,360
precious so making um good decisions

120
00:05:29,360 --> 00:05:34,360
fast is a key to success so looking in h

121
00:05:32,360 --> 00:05:37,680
side I can confidently say that choosing

122
00:05:34,360 --> 00:05:39,639
Jango was the right decision um Jango is

123
00:05:37,680 --> 00:05:41,880
a great framework and one of the

124
00:05:39,639 --> 00:05:44,360
features that proved to be very useful

125
00:05:41,880 --> 00:05:48,000
for us is support for multiple databases

126
00:05:44,360 --> 00:05:50,680
and it is also the topic of my today's

127
00:05:48,000 --> 00:05:53,680
talk I will start with explaining our

128
00:05:50,680 --> 00:05:55,360
motivation for using multiple databases

129
00:05:53,680 --> 00:05:58,680
in

130
00:05:55,360 --> 00:06:01,319
nextcore as I mentioned in 2021 we had a

131
00:05:58,680 --> 00:06:03,960
a few new large Partners integrating

132
00:06:01,319 --> 00:06:05,479
with xcore and several new verticals and

133
00:06:03,960 --> 00:06:08,560
some of these

134
00:06:05,479 --> 00:06:10,199
Partners had quite strict uh

135
00:06:08,560 --> 00:06:11,800
requirements for througho and response

136
00:06:10,199 --> 00:06:14,479
time and considering the complexity of

137
00:06:11,800 --> 00:06:16,400
the business logic um it was not a

138
00:06:14,479 --> 00:06:19,000
trivial problem to scale this

139
00:06:16,400 --> 00:06:20,479
application um the computer resources

140
00:06:19,000 --> 00:06:22,599
were not a problem the application was

141
00:06:20,479 --> 00:06:24,520
running on kubernetes cluster and was

142
00:06:22,599 --> 00:06:26,599
able to scale horizontally using

143
00:06:24,520 --> 00:06:29,280
horizontal Port Auto scaler however the

144
00:06:26,599 --> 00:06:31,039
bottleneck was the database uh we didn't

145
00:06:29,280 --> 00:06:33,160
have another way of scaling it apart

146
00:06:31,039 --> 00:06:34,599
from increasing the size but um it would

147
00:06:33,160 --> 00:06:37,400
be a costly

148
00:06:34,599 --> 00:06:40,560
solution um and more we still could

149
00:06:37,400 --> 00:06:43,120
potentially hit certain limits um for

150
00:06:40,560 --> 00:06:45,479
example dis copes um so we started

151
00:06:43,120 --> 00:06:48,440
working on optimizing database workloads

152
00:06:45,479 --> 00:06:51,639
we notice that um quot operation

153
00:06:48,440 --> 00:06:54,560
contributed around 85% of all sccore

154
00:06:51,639 --> 00:06:56,720
traffic and it gave us the narrow place

155
00:06:54,560 --> 00:06:58,879
we could focus on WE analyzed the theb

156
00:06:56,720 --> 00:07:01,039
work CLS generated by code operation and

157
00:06:58,879 --> 00:07:03,160
noticed that from the DB perspective

158
00:07:01,039 --> 00:07:05,560
there are two large steps in the

159
00:07:03,160 --> 00:07:09,080
workflow in the first step the operation

160
00:07:05,560 --> 00:07:10,520
would uh read various uh configurations

161
00:07:09,080 --> 00:07:12,960
from database such as product

162
00:07:10,520 --> 00:07:15,639
configurations product selection rules

163
00:07:12,960 --> 00:07:18,160
partner configurations and so on and

164
00:07:15,639 --> 00:07:20,520
this data then used to generate quote

165
00:07:18,160 --> 00:07:22,759
objects which are calculated and

166
00:07:20,520 --> 00:07:28,039
recorded back in the

167
00:07:22,759 --> 00:07:29,800
database uh and um this quotes table uh

168
00:07:28,039 --> 00:07:33,000
was the part of the same database where

169
00:07:29,800 --> 00:07:34,759
the configuration data was stored so

170
00:07:33,000 --> 00:07:37,639
apart from quotes we also had other

171
00:07:34,759 --> 00:07:40,840
types of generated customer data um such

172
00:07:37,639 --> 00:07:44,759
as transactions uh customers uh policies

173
00:07:40,840 --> 00:07:47,199
Etc so the 10 tables in the database um

174
00:07:44,759 --> 00:07:49,960
stored a several terabyte of data

175
00:07:47,199 --> 00:07:53,400
whereas the rest 60 tables stored only

176
00:07:49,960 --> 00:07:55,840
100 Megs of um configuration data in

177
00:07:53,400 --> 00:07:57,400
total um so that was the the the reason

178
00:07:55,840 --> 00:07:59,479
number one we saw if we could use a

179
00:07:57,400 --> 00:08:01,240
separate database to store configur

180
00:07:59,479 --> 00:08:03,240
option we could potentially aflo a

181
00:08:01,240 --> 00:08:06,240
significant amount of read operations to

182
00:08:03,240 --> 00:08:08,960
this uh new database increasing overall

183
00:08:06,240 --> 00:08:08,960
capacity of the

184
00:08:09,319 --> 00:08:14,560
system um the other important

185
00:08:11,639 --> 00:08:17,000
consideration was the future ability to

186
00:08:14,560 --> 00:08:19,199
implement Regional deployments as cover

187
00:08:17,000 --> 00:08:21,919
genius uh is a global business it was

188
00:08:19,199 --> 00:08:24,000
quite important for us uh to be able to

189
00:08:21,919 --> 00:08:26,280
store customer data in different

190
00:08:24,000 --> 00:08:27,479
locations um as required by different

191
00:08:26,280 --> 00:08:31,560
insurance

192
00:08:27,479 --> 00:08:33,320
regulations um so extracting

193
00:08:31,560 --> 00:08:35,880
configurations into a separate database

194
00:08:33,320 --> 00:08:37,680
also enable the team to implement uh

195
00:08:35,880 --> 00:08:40,440
Regional deployments while maintaining

196
00:08:37,680 --> 00:08:42,800
one centralized set of configurations

197
00:08:40,440 --> 00:08:45,360
for uh all

198
00:08:42,800 --> 00:08:47,480
regions finally splitting database would

199
00:08:45,360 --> 00:08:49,600
make it much easier for us to add more

200
00:08:47,480 --> 00:08:51,760
uh read replicas uh of configurations

201
00:08:49,600 --> 00:08:53,200
database if necessary and increase

202
00:08:51,760 --> 00:08:57,000
capacity in this

203
00:08:53,200 --> 00:08:59,920
way um so yeah this is what we did

204
00:08:57,000 --> 00:09:02,320
slightly later um using multiple

205
00:08:59,920 --> 00:09:04,720
replicas helped us to increase capacity

206
00:09:02,320 --> 00:09:06,800
and improve availability of the

207
00:09:04,720 --> 00:09:10,399
system all right now I'm moving to the

208
00:09:06,800 --> 00:09:12,800
main part of my talk um by the time when

209
00:09:10,399 --> 00:09:14,440
we realized the need for these changes

210
00:09:12,800 --> 00:09:17,320
we already had quite significant volume

211
00:09:14,440 --> 00:09:20,160
of traffic in production um excore was

212
00:09:17,320 --> 00:09:24,720
already serving million millions of API

213
00:09:20,160 --> 00:09:26,480
requests uh per day um and uh in this

214
00:09:24,720 --> 00:09:28,839
part of this talk I will speak about the

215
00:09:26,480 --> 00:09:30,600
step-by-step process we took to split

216
00:09:28,839 --> 00:09:34,040
the database

217
00:09:30,600 --> 00:09:36,760
two so the step number one was to plan

218
00:09:34,040 --> 00:09:39,200
everything uh it was important to

219
00:09:36,760 --> 00:09:41,240
analyze the tables to understand better

220
00:09:39,200 --> 00:09:43,680
the workloads data flows and

221
00:09:41,240 --> 00:09:45,680
dependencies between models we reviewed

222
00:09:43,680 --> 00:09:48,440
every table and split them in two

223
00:09:45,680 --> 00:09:51,240
categories configurations and customer

224
00:09:48,440 --> 00:09:53,519
data the diagram on this slide is a

225
00:09:51,240 --> 00:09:56,600
really simplified example but in reality

226
00:09:53,519 --> 00:09:59,440
we had approximately 70 different tables

227
00:09:56,600 --> 00:10:01,160
um so it was a bit more complicated

228
00:09:59,440 --> 00:10:02,880
uh we had to analyze every single one of

229
00:10:01,160 --> 00:10:04,440
them and make that

230
00:10:02,880 --> 00:10:09,200
decision

231
00:10:04,440 --> 00:10:11,440
um so the next step was to remove uh

232
00:10:09,200 --> 00:10:13,880
existing foreign key constraints in the

233
00:10:11,440 --> 00:10:17,120
database and uh

234
00:10:13,880 --> 00:10:19,279
Jango um can support foreign keys that

235
00:10:17,120 --> 00:10:21,440
refer to a table in another

236
00:10:19,279 --> 00:10:23,040
database perfectly of course there are

237
00:10:21,440 --> 00:10:25,440
certain limitations with this approach

238
00:10:23,040 --> 00:10:27,760
for example uh joining data from

239
00:10:25,440 --> 00:10:30,399
different databases in a single quer set

240
00:10:27,760 --> 00:10:33,200
won't be possible

241
00:10:30,399 --> 00:10:36,000
uh but the cool thing is that most ofm

242
00:10:33,200 --> 00:10:38,320
functionality would still work um the

243
00:10:36,000 --> 00:10:41,920
only required change is to set this uh

244
00:10:38,320 --> 00:10:43,639
DB constraint uh option to false as you

245
00:10:41,920 --> 00:10:45,240
can see in this

246
00:10:43,639 --> 00:10:48,040
slide

247
00:10:45,240 --> 00:10:50,800
um we were not worried about data

248
00:10:48,040 --> 00:10:53,480
consistency as all of the configurations

249
00:10:50,800 --> 00:10:54,880
are immutable so that was a a simple and

250
00:10:53,480 --> 00:10:56,680
safe change that we could deploy in

251
00:10:54,880 --> 00:10:59,440
production straight

252
00:10:56,680 --> 00:11:01,279
away uh now in production we had all the

253
00:10:59,440 --> 00:11:03,440
tables divided into separate categories

254
00:11:01,279 --> 00:11:05,040
without any relations between uh between

255
00:11:03,440 --> 00:11:08,000
the two

256
00:11:05,040 --> 00:11:10,600
categories um so the next step was to

257
00:11:08,000 --> 00:11:12,680
set up a local development environment

258
00:11:10,600 --> 00:11:14,800
um at cover genius we use Docker compost

259
00:11:12,680 --> 00:11:17,160
for local development and we didn't want

260
00:11:14,800 --> 00:11:18,920
to add another uh postgress container in

261
00:11:17,160 --> 00:11:21,560
the local environment so we just decided

262
00:11:18,920 --> 00:11:23,600
to use uh logical logical databases

263
00:11:21,560 --> 00:11:25,040
inside the same postgress container so

264
00:11:23,600 --> 00:11:28,560
this is an example of script that

265
00:11:25,040 --> 00:11:29,800
creates uh multiple uh uh databases for

266
00:11:28,560 --> 00:11:32,560
us

267
00:11:29,800 --> 00:11:35,079
um and uh to start using the script uh

268
00:11:32,560 --> 00:11:37,079
we only need to mount it into correct

269
00:11:35,079 --> 00:11:39,959
directory and set the required uh

270
00:11:37,079 --> 00:11:42,720
environment variable and well we could

271
00:11:39,959 --> 00:11:45,040
use uh multiple databases in the in

272
00:11:42,720 --> 00:11:45,040
local

273
00:11:45,200 --> 00:11:50,200
environment um now we need to instruct

274
00:11:48,240 --> 00:11:52,839
Jango on how to connect uh to the

275
00:11:50,200 --> 00:11:56,040
individual databases it's pretty easy to

276
00:11:52,839 --> 00:11:57,760
do uh we just need to add one uh new

277
00:11:56,040 --> 00:12:00,440
connection to the databases dictionary

278
00:11:57,760 --> 00:12:03,560
in our settings.py this is not a risky

279
00:12:00,440 --> 00:12:05,720
change either and can be easily um

280
00:12:03,560 --> 00:12:07,399
deployed in production immediately and

281
00:12:05,720 --> 00:12:10,040
the reason for this is uh by default

282
00:12:07,399 --> 00:12:14,600
junga assumes that only default database

283
00:12:10,040 --> 00:12:16,199
connection exist um and to start uh

284
00:12:14,600 --> 00:12:18,959
using the new connection in junko there

285
00:12:16,199 --> 00:12:22,279
are two ways the first one is uh by

286
00:12:18,959 --> 00:12:25,839
explicitly calling um a using method on

287
00:12:22,279 --> 00:12:28,880
query set object and the second option

288
00:12:25,839 --> 00:12:32,720
uh is to delegate that decision to a DB

289
00:12:28,880 --> 00:12:36,560
r about her um in a large application

290
00:12:32,720 --> 00:12:39,040
the second option is um probably always

291
00:12:36,560 --> 00:12:41,839
better um as it requires less code

292
00:12:39,040 --> 00:12:43,800
changes and it also probably more

293
00:12:41,839 --> 00:12:47,440
maintainable

294
00:12:43,800 --> 00:12:49,760
too um so the next step was to add a DB

295
00:12:47,440 --> 00:12:52,480
router you haven't used DB routers in

296
00:12:49,760 --> 00:12:55,839
junga before no worries it's pretty easy

297
00:12:52,480 --> 00:12:58,079
um DB router is a simple class uh that

298
00:12:55,839 --> 00:13:00,000
makes it possible to hook into internal

299
00:12:58,079 --> 00:13:03,360
orm logic

300
00:13:00,000 --> 00:13:04,079
and instruct jungo omm on how to route

301
00:13:03,360 --> 00:13:06,720
the

302
00:13:04,079 --> 00:13:08,680
queries let's have a little bit deeper

303
00:13:06,720 --> 00:13:12,440
look at each of these

304
00:13:08,680 --> 00:13:15,639
methods so firstly we need to decide how

305
00:13:12,440 --> 00:13:17,920
to route read and write queries both DB

306
00:13:15,639 --> 00:13:21,320
for read and DB for write methods uh

307
00:13:17,920 --> 00:13:23,160
take this um uh model class as an

308
00:13:21,320 --> 00:13:25,000
argument and uh return the name of

309
00:13:23,160 --> 00:13:29,199
database connection to

310
00:13:25,000 --> 00:13:30,880
use um and uh if we need to store um

311
00:13:29,199 --> 00:13:32,760
different models in different databases

312
00:13:30,880 --> 00:13:35,440
one simple way of implementing this is

313
00:13:32,760 --> 00:13:37,600
through uh maintaining a mapping between

314
00:13:35,440 --> 00:13:40,639
the model name and the connection name

315
00:13:37,600 --> 00:13:42,399
to use um so this piece of code

316
00:13:40,639 --> 00:13:45,279
demonstrates potential implementation

317
00:13:42,399 --> 00:13:48,000
but our actual implementation was more

318
00:13:45,279 --> 00:13:50,560
complex uh for example we can Define the

319
00:13:48,000 --> 00:13:52,519
databases per uh jungo application or

320
00:13:50,560 --> 00:13:55,079
per uh individual

321
00:13:52,519 --> 00:13:57,720
model we also run some checks on Startup

322
00:13:55,079 --> 00:13:59,680
to ensure that each model um in the

323
00:13:57,720 --> 00:14:02,480
application is explicitly assigned to

324
00:13:59,680 --> 00:14:06,279
one of the two

325
00:14:02,480 --> 00:14:08,560
databases so next we need to um instruct

326
00:14:06,279 --> 00:14:10,759
the router if it should allow relations

327
00:14:08,560 --> 00:14:13,079
between the two objects uh allow

328
00:14:10,759 --> 00:14:16,320
relation is simply a validation method

329
00:14:13,079 --> 00:14:19,279
that is used inside um inside jungo

330
00:14:16,320 --> 00:14:21,720
framework in a few places uh but as we

331
00:14:19,279 --> 00:14:24,320
store individual models in separate

332
00:14:21,720 --> 00:14:26,680
databases uh we need this method to

333
00:14:24,320 --> 00:14:29,360
explicitly return true otherwise we

334
00:14:26,680 --> 00:14:33,399
would be uh wouldn't be able to use that

335
00:14:29,360 --> 00:14:35,639
was um uh using foreign keys with DB

336
00:14:33,399 --> 00:14:39,639
constraint

337
00:14:35,639 --> 00:14:41,360
fals um so the last problem to the

338
00:14:39,639 --> 00:14:43,759
router solve is to decide what

339
00:14:41,360 --> 00:14:47,800
migrations should be applied to what

340
00:14:43,759 --> 00:14:52,000
databases the um allow migrate method is

341
00:14:47,800 --> 00:14:54,720
um used um just um for this uh it's

342
00:14:52,000 --> 00:14:57,160
called for every migration operation and

343
00:14:54,720 --> 00:14:58,639
uh returns a Boolean flag indicating if

344
00:14:57,160 --> 00:15:01,839
the migration should be applied to the

345
00:14:58,639 --> 00:15:03,680
given database when multiple databases

346
00:15:01,839 --> 00:15:04,399
are used to store different models we

347
00:15:03,680 --> 00:15:08,000
need

348
00:15:04,399 --> 00:15:09,079
to uh run my great command for each of

349
00:15:08,000 --> 00:15:11,800
them

350
00:15:09,079 --> 00:15:15,000
separately um and the potential

351
00:15:11,800 --> 00:15:17,000
implementation is on this slide but uh

352
00:15:15,000 --> 00:15:19,800
but please not this special handling for

353
00:15:17,000 --> 00:15:21,560
use case when both connections refer

354
00:15:19,800 --> 00:15:24,800
point to the same database

355
00:15:21,560 --> 00:15:26,920
instance um J needs a separate

356
00:15:24,800 --> 00:15:29,839
migrations table in each database it

357
00:15:26,920 --> 00:15:33,199
manages to track uh both migrations are

358
00:15:29,839 --> 00:15:35,279
applied and and we when we point both

359
00:15:33,199 --> 00:15:38,199
connections to the same database there

360
00:15:35,279 --> 00:15:40,959
is only one migration stable uh and all

361
00:15:38,199 --> 00:15:43,759
of all of the changes to all models must

362
00:15:40,959 --> 00:15:46,800
be stored there uh so the correct way of

363
00:15:43,759 --> 00:15:49,720
handling is to apply all migration and

364
00:15:46,800 --> 00:15:54,319
this is what uh the return true part

365
00:15:49,720 --> 00:15:55,800
does here um when migrate command is

366
00:15:54,319 --> 00:15:57,600
called on the second database connection

367
00:15:55,800 --> 00:16:00,000
it will just happily SK skip all the

368
00:15:57,600 --> 00:16:02,079
migrations as the do already presented

369
00:16:00,000 --> 00:16:03,600
in the migrations

370
00:16:02,079 --> 00:16:06,279
table

371
00:16:03,600 --> 00:16:09,000
um one more caveat is for using run

372
00:16:06,279 --> 00:16:12,360
Python and run SQL methods inside

373
00:16:09,000 --> 00:16:14,959
migrations in this case um model

374
00:16:12,360 --> 00:16:17,759
name uh argument passed in the allow

375
00:16:14,959 --> 00:16:20,120
migrate method will uh contain none and

376
00:16:17,759 --> 00:16:22,759
the only way for the DB router to know

377
00:16:20,120 --> 00:16:26,920
what database um to use in this case is

378
00:16:22,759 --> 00:16:29,920
to to instruct it explicitly by uh using

379
00:16:26,920 --> 00:16:29,920
hints

380
00:16:31,000 --> 00:16:37,399
um so yeah we had to review and PCH all

381
00:16:34,360 --> 00:16:39,079
run Python and run SQL migration uh

382
00:16:37,399 --> 00:16:41,800
operations and ensure all of them have

383
00:16:39,079 --> 00:16:44,959
correct hint argument provided uh

384
00:16:41,800 --> 00:16:48,240
something like on this

385
00:16:44,959 --> 00:16:50,120
slide uh now once DB router is ready we

386
00:16:48,240 --> 00:16:52,120
just need to enable it in the

387
00:16:50,120 --> 00:16:54,440
configuration we need to specify it in

388
00:16:52,120 --> 00:16:57,279
the database router setting and as you

389
00:16:54,440 --> 00:16:59,560
can see um it is possible to use

390
00:16:57,279 --> 00:17:02,279
multiple DB routers

391
00:16:59,560 --> 00:17:04,679
um and junga can chain those calls if

392
00:17:02,279 --> 00:17:07,160
required and while I could think of a

393
00:17:04,679 --> 00:17:08,919
potential use case for this I think it's

394
00:17:07,160 --> 00:17:11,319
probably cleaner and easier to maintain

395
00:17:08,919 --> 00:17:14,520
a single DB router for all models and

396
00:17:11,319 --> 00:17:19,319
this is how we implemented this INX core

397
00:17:14,520 --> 00:17:22,439
to um so now introducing DB router is a

398
00:17:19,319 --> 00:17:24,600
is a very significant change uh be very

399
00:17:22,439 --> 00:17:26,640
careful and expect things to break if

400
00:17:24,600 --> 00:17:28,520
you don't have a good code coverage I

401
00:17:26,640 --> 00:17:29,400
would recommend you not not do this at

402
00:17:28,520 --> 00:17:31,200
all

403
00:17:29,400 --> 00:17:35,559
uh we had a very good test coverage for

404
00:17:31,200 --> 00:17:39,200
excore um all critical operations um

405
00:17:35,559 --> 00:17:42,760
were covered 100% and uh we had around

406
00:17:39,200 --> 00:17:44,880
3,000 tests uh back then um and around

407
00:17:42,760 --> 00:17:46,000
80% of the tests that started failing

408
00:17:44,880 --> 00:17:49,280
after this

409
00:17:46,000 --> 00:17:51,919
change luckily it was easy fix um for

410
00:17:49,280 --> 00:17:54,919
many of them as um you know um it was a

411
00:17:51,919 --> 00:17:56,840
simple configuration change um so by

412
00:17:54,919 --> 00:17:59,360
default Jenga assumes that only default

413
00:17:56,840 --> 00:18:00,720
database is used um

414
00:17:59,360 --> 00:18:05,240
uh and

415
00:18:00,720 --> 00:18:06,600
uh testing framework that um you know

416
00:18:05,240 --> 00:18:08,200
and basically when you test the code

417
00:18:06,600 --> 00:18:10,720
that requires multiple databases it

418
00:18:08,200 --> 00:18:12,919
requires a special handling we need to

419
00:18:10,720 --> 00:18:16,400
explicitly instruct jungo testing

420
00:18:12,919 --> 00:18:19,919
framework to run um migrations on

421
00:18:16,400 --> 00:18:22,640
multiple databases so um in excore we

422
00:18:19,919 --> 00:18:25,039
use both junga sty uh test cases and P

423
00:18:22,640 --> 00:18:30,240
test with jungo test plugin and luckily

424
00:18:25,039 --> 00:18:34,159
it it was um a simple change for uh both

425
00:18:30,240 --> 00:18:36,559
setups um the other reason for fail test

426
00:18:34,159 --> 00:18:39,559
was the usage of Select related

427
00:18:36,559 --> 00:18:42,400
optimization um so storing models in

428
00:18:39,559 --> 00:18:47,200
individual databases uh we cannot use uh

429
00:18:42,400 --> 00:18:49,960
select um so select real is not usable

430
00:18:47,200 --> 00:18:53,919
anymore uh this op optimization operates

431
00:18:49,960 --> 00:18:56,600
at SQL level and the tables um they they

432
00:18:53,919 --> 00:18:59,320
they live in separate databases um and

433
00:18:56,600 --> 00:19:01,720
it is not quite possible to use as

434
00:18:59,320 --> 00:19:03,760
joints so instead of Select related we

435
00:19:01,720 --> 00:19:05,640
switched to prefetch related where it

436
00:19:03,760 --> 00:19:09,200
was possible or we did some other

437
00:19:05,640 --> 00:19:09,200
refactoring to achieve the required

438
00:19:10,240 --> 00:19:17,360
performance finally the last group of uh

439
00:19:12,840 --> 00:19:20,080
failed tests uh was caused by um using

440
00:19:17,360 --> 00:19:23,120
transactions uh the transaction methods

441
00:19:20,080 --> 00:19:25,600
such as atomic set roll back and uh on

442
00:19:23,120 --> 00:19:28,320
Commit uh they all implicitly use

443
00:19:25,600 --> 00:19:30,280
default database um and for excore we

444
00:19:28,320 --> 00:19:32,360
decided that default database will be

445
00:19:30,280 --> 00:19:34,919
the one that that stores uh

446
00:19:32,360 --> 00:19:37,720
configurations and in most uh of the

447
00:19:34,919 --> 00:19:43,280
views we uh it was the not the desire

448
00:19:37,720 --> 00:19:46,600
desired logic so uh we had to change it

449
00:19:43,280 --> 00:19:49,440
um luckily that was the the last change

450
00:19:46,600 --> 00:19:52,559
after pushing it into

451
00:19:49,440 --> 00:19:55,640
um gitlab we could see we could we

452
00:19:52,559 --> 00:19:57,760
started seeing green uh CIP plants it

453
00:19:55,640 --> 00:20:00,520
was the time to test uh this in staging

454
00:19:57,760 --> 00:20:02,840
environment

455
00:20:00,520 --> 00:20:06,760
so deploy deploying the change was not

456
00:20:02,840 --> 00:20:09,159
overly complex um we decided it wasn't

457
00:20:06,760 --> 00:20:11,600
worse to automate end to endend as it's

458
00:20:09,159 --> 00:20:13,760
uh just a one-off operation so we

459
00:20:11,600 --> 00:20:16,760
originally deployed all the router call

460
00:20:13,760 --> 00:20:18,240
changes but made uh both connections uh

461
00:20:16,760 --> 00:20:19,240
pointing to the same database as I

462
00:20:18,240 --> 00:20:21,720
already

463
00:20:19,240 --> 00:20:24,840
mentioned and

464
00:20:21,720 --> 00:20:27,640
um um yeah and after this we posted all

465
00:20:24,840 --> 00:20:30,760
rights to the configuration uh tables in

466
00:20:27,640 --> 00:20:32,760
the old database and created a dump of

467
00:20:30,760 --> 00:20:36,280
the required tables which we loaded into

468
00:20:32,760 --> 00:20:38,080
a new database um to avoid mistakes we

469
00:20:36,280 --> 00:20:41,520
created a jungle management command that

470
00:20:38,080 --> 00:20:43,799
generates um correct PG dump statements

471
00:20:41,520 --> 00:20:46,240
with all the required uh taable

472
00:20:43,799 --> 00:20:47,960
specified and after this we updated

473
00:20:46,240 --> 00:20:50,960
database

474
00:20:47,960 --> 00:20:52,919
configurations uh to use uh connections

475
00:20:50,960 --> 00:20:54,600
to different database and that was it

476
00:20:52,919 --> 00:20:57,039
after testing the deployment process in

477
00:20:54,600 --> 00:20:59,440
staging uh we repeated the same uh

478
00:20:57,039 --> 00:21:01,120
process in production and we had zero

479
00:20:59,440 --> 00:21:03,440
down time throughout the process and no

480
00:21:01,120 --> 00:21:03,440
data

481
00:21:03,799 --> 00:21:09,600
loss uh later we implemented support for

482
00:21:07,640 --> 00:21:12,880
read read

483
00:21:09,600 --> 00:21:16,200
replicas um it was also an interesting

484
00:21:12,880 --> 00:21:18,960
project but um I will leave this topic

485
00:21:16,200 --> 00:21:20,799
for another day um on this slide I would

486
00:21:18,960 --> 00:21:23,520
like to present the final database

487
00:21:20,799 --> 00:21:26,320
architecture that we've been using until

488
00:21:23,520 --> 00:21:28,279
last year um and last year we made

489
00:21:26,320 --> 00:21:31,279
further changes we started using no

490
00:21:28,279 --> 00:21:34,440
scale database for uh storing CS

491
00:21:31,279 --> 00:21:36,600
data and uh next year we consider even

492
00:21:34,440 --> 00:21:40,400
more changes as the number of AP API

493
00:21:36,600 --> 00:21:43,000
consumers uh continues to grow and we

494
00:21:40,400 --> 00:21:45,440
discussing such topics as isolating

495
00:21:43,000 --> 00:21:47,799
sensitive data into a separate database

496
00:21:45,440 --> 00:21:51,919
and um also

497
00:21:47,799 --> 00:21:51,919
introducing one more for data

498
00:21:53,120 --> 00:21:59,640
archives all right um what did I want to

499
00:21:57,120 --> 00:22:02,200
achieve with my talk

500
00:21:59,640 --> 00:22:03,600
um first I wanted to share our

501
00:22:02,200 --> 00:22:05,760
experience of scaling the jungo

502
00:22:03,600 --> 00:22:08,200
applications by adding multiple

503
00:22:05,760 --> 00:22:09,480
databases potentially it will be useful

504
00:22:08,200 --> 00:22:12,600
for someone

505
00:22:09,480 --> 00:22:15,320
else um second I wanted to showcase the

506
00:22:12,600 --> 00:22:18,200
junga um the junga functionality for

507
00:22:15,320 --> 00:22:20,520
supporting multiple databases and I also

508
00:22:18,200 --> 00:22:22,559
want to use this opportunity to say

509
00:22:20,520 --> 00:22:25,039
thank you to all the amazing junga

510
00:22:22,559 --> 00:22:27,559
contributors who made this framework so

511
00:22:25,039 --> 00:22:29,360
awesome and also to the Instagram

512
00:22:27,559 --> 00:22:32,240
Engineers who I think Pioneer using

513
00:22:29,360 --> 00:22:35,039
junga for high loaded

514
00:22:32,240 --> 00:22:37,000
applications my third goal was to make

515
00:22:35,039 --> 00:22:39,600
um you think more proactively about the

516
00:22:37,000 --> 00:22:42,360
database architecture in your own

517
00:22:39,600 --> 00:22:46,400
projects uh and maybe my talk could give

518
00:22:42,360 --> 00:22:48,480
you some clue or um QE or

519
00:22:46,400 --> 00:22:51,520
inspiration uh but remember using

520
00:22:48,480 --> 00:22:53,720
multiple databases always um almost

521
00:22:51,520 --> 00:22:56,039
always imply additional complexity in

522
00:22:53,720 --> 00:22:59,600
your application so this complexity

523
00:22:56,039 --> 00:23:01,320
needs to be justified uh uh for simpler

524
00:22:59,600 --> 00:23:03,679
cases consider using other ways of

525
00:23:01,320 --> 00:23:06,520
scaling database for example using fully

526
00:23:03,679 --> 00:23:08,279
managed DB cluster RDS multiz cluster

527
00:23:06,520 --> 00:23:11,679
deployments uh for

528
00:23:08,279 --> 00:23:13,640
example um and if you would like to ask

529
00:23:11,679 --> 00:23:15,600
further question about this or related

530
00:23:13,640 --> 00:23:17,960
topic please don't hesitate connecting

531
00:23:15,600 --> 00:23:21,760
with me on LinkedIn uh I would love to

532
00:23:17,960 --> 00:23:23,559
help if I can um in the end I want to

533
00:23:21,760 --> 00:23:25,440
say thank you to my amazing team at

534
00:23:23,559 --> 00:23:28,520
cover genius it's been great pleasure to

535
00:23:25,440 --> 00:23:29,919
work with this team over the years and I

536
00:23:28,520 --> 00:23:31,600
wouldn't be here presenting this topic

537
00:23:29,919 --> 00:23:34,120
today without my

538
00:23:31,600 --> 00:23:36,300
team thanks to all who listened my talk

539
00:23:34,120 --> 00:23:43,130
today I'm happy to answer questions if

540
00:23:36,300 --> 00:23:43,130
[Applause]

541
00:23:49,000 --> 00:23:55,720
any okay uh thank you and now we have

542
00:23:52,640 --> 00:23:59,240
some time for questions so uh if anyone

543
00:23:55,720 --> 00:24:03,320
would like to answer question please uh

544
00:23:59,240 --> 00:24:03,320
put your hands up

545
00:24:07,679 --> 00:24:13,279
excellent hello and thanks for talk so

546
00:24:11,360 --> 00:24:19,120
previously I heard you mention you have

547
00:24:13,279 --> 00:24:20,840
like three, tests for this uh Dango

548
00:24:19,120 --> 00:24:23,200
application so how do you write this

549
00:24:20,840 --> 00:24:24,960
test did you use some autogeneration

550
00:24:23,200 --> 00:24:28,480
tools for writing this

551
00:24:24,960 --> 00:24:31,840
test sorry how do I do yeah because you

552
00:24:28,480 --> 00:24:33,960
you have three, test so how did you

553
00:24:31,840 --> 00:24:37,440
write this test did you write them by

554
00:24:33,960 --> 00:24:39,840
hand manually or you use some generation

555
00:24:37,440 --> 00:24:42,600
tools for that yeah we've been working

556
00:24:39,840 --> 00:24:46,279
on this application for seven years now

557
00:24:42,600 --> 00:24:48,240
so you know um so four years ago I think

558
00:24:46,279 --> 00:24:49,799
these days it's around 4 and a half

559
00:24:48,240 --> 00:24:54,600
thousand tests yeah we write them

560
00:24:49,799 --> 00:24:57,960
manually um um yes we use uh some

561
00:24:54,600 --> 00:24:59,760
parameterized test so um I'm actually

562
00:24:57,960 --> 00:25:02,720
not sure if this is number it's probably

563
00:24:59,760 --> 00:25:06,720
number with all the different variations

564
00:25:02,720 --> 00:25:06,720
uh that that run as well

565
00:25:15,760 --> 00:25:21,399
yeah um thank you uh just a question on

566
00:25:18,880 --> 00:25:24,679
any thoughts on um indexing and

567
00:25:21,399 --> 00:25:26,200
partitioning you know um yeah because uh

568
00:25:24,679 --> 00:25:27,880
we I think we are in the same situation

569
00:25:26,200 --> 00:25:30,600
as well so you know I'm I I need to

570
00:25:27,880 --> 00:25:34,039
explore for um and you know get some

571
00:25:30,600 --> 00:25:38,720
advice on partitioning in Jango

572
00:25:34,039 --> 00:25:41,039
mhm um yes so

573
00:25:38,720 --> 00:25:44,640
partitioning yeah we we explored options

574
00:25:41,039 --> 00:25:46,520
with partitioning as well um basically

575
00:25:44,640 --> 00:25:51,080
um you know the data that we need stor

576
00:25:46,520 --> 00:25:55,640
is probably um still can fit in one uh

577
00:25:51,080 --> 00:25:57,240
DB uh server so we because partitioning

578
00:25:55,640 --> 00:25:59,760
would mean that you

579
00:25:57,240 --> 00:26:02,640
would need to increase the complexity of

580
00:25:59,760 --> 00:26:04,679
your app even further so for now we uh

581
00:26:02,640 --> 00:26:07,360
decided not to do this unless it's

582
00:26:04,679 --> 00:26:09,399
absolutely necessary so we have a way of

583
00:26:07,360 --> 00:26:12,080
uh removing all data from the database

584
00:26:09,399 --> 00:26:14,919
so there is a um you know process to

585
00:26:12,080 --> 00:26:18,240
clean up um you know some of the data

586
00:26:14,919 --> 00:26:20,440
that uh we don't need anymore um but um

587
00:26:18,240 --> 00:26:21,760
yeah I think these days probably we have

588
00:26:20,440 --> 00:26:25,520
around

589
00:26:21,760 --> 00:26:29,399
um 10 to 12 terab of data and we also

590
00:26:25,520 --> 00:26:34,080
considering uh switching from posr SQL

591
00:26:29,399 --> 00:26:35,480
to Aurora uh for um yeah but probably

592
00:26:34,080 --> 00:26:39,200
don't want to do

593
00:26:35,480 --> 00:26:39,200
practitioning this is what the team

594
00:26:39,880 --> 00:26:46,360
decided good day um just wondering if

595
00:26:43,559 --> 00:26:49,360
you'd done any benchmarking and thinking

596
00:26:46,360 --> 00:26:51,799
of the keynote particularly um looking

597
00:26:49,360 --> 00:26:54,039
at what uh how your performance was

598
00:26:51,799 --> 00:26:57,200
before you did the change how how it's

599
00:26:54,039 --> 00:26:58,799
gone afterwards what things made you do

600
00:26:57,200 --> 00:27:00,559
things like the read replic

601
00:26:58,799 --> 00:27:03,600
because

602
00:27:00,559 --> 00:27:05,480
um yeah we uh we have

603
00:27:03,600 --> 00:27:08,320
a

604
00:27:05,480 --> 00:27:10,440
um we have performance testing uh

605
00:27:08,320 --> 00:27:13,480
framework we use

606
00:27:10,440 --> 00:27:17,679
um oh this

607
00:27:13,480 --> 00:27:19,279
days D just vanished from my head

608
00:27:17,679 --> 00:27:22,559
this

609
00:27:19,279 --> 00:27:24,440
um yes

610
00:27:22,559 --> 00:27:26,600
um

611
00:27:24,440 --> 00:27:28,279
uh yeah we have we have a suite of

612
00:27:26,600 --> 00:27:31,679
performance test that we that we still

613
00:27:28,279 --> 00:27:36,360
run uh we don't we don't run uh them as

614
00:27:31,679 --> 00:27:38,320
a part of CI CD pipeline it's a um we

615
00:27:36,360 --> 00:27:42,000
have automation we on this uh some of

616
00:27:38,320 --> 00:27:46,360
the tests were on weekly

617
00:27:42,000 --> 00:27:46,360
um um yeah and

618
00:27:47,080 --> 00:27:54,760
um um The Benchmark I mentioned it was

619
00:27:50,919 --> 00:27:57,840
even before we choose the web framework

620
00:27:54,760 --> 00:27:59,760
uh it was a seven years ago I can't

621
00:27:57,840 --> 00:28:04,399
remember again like what the framework

622
00:27:59,760 --> 00:28:07,120
we use but um but yeah I yeah we can we

623
00:28:04,399 --> 00:28:09,799
can stay after the talk and I yeah I can

624
00:28:07,120 --> 00:28:12,159
speak more about this

625
00:28:09,799 --> 00:28:15,840
yeah okay do we have any further

626
00:28:12,159 --> 00:28:15,840
questions yes we have a couple

627
00:28:18,200 --> 00:28:23,159
excellent um if you're if you're willing

628
00:28:20,440 --> 00:28:25,720
to say on on on the record uh how long

629
00:28:23,159 --> 00:28:28,320
do your 4,500 tests take to

630
00:28:25,720 --> 00:28:32,519
run okay looks like everyone is

631
00:28:28,320 --> 00:28:36,519
interested in testing topic um yeah we

632
00:28:32,519 --> 00:28:41,159
we we run them in um I think four uh

633
00:28:36,519 --> 00:28:43,399
parallel um CI jobs uh now um and uh I

634
00:28:41,159 --> 00:28:47,640
think it's like 15 minutes something

635
00:28:43,399 --> 00:28:47,640
like this for not not too

636
00:28:51,279 --> 00:28:57,799
long cheers um yeah thank you it's a

637
00:28:54,120 --> 00:29:00,159
really interesting idea um but I can

638
00:28:57,799 --> 00:29:02,480
imagine that like losing that ability to

639
00:29:00,159 --> 00:29:05,200
kind of do joins across uh different

640
00:29:02,480 --> 00:29:07,960
tables in the would be perhaps a little

641
00:29:05,200 --> 00:29:12,080
limiting have you found that to be a

642
00:29:07,960 --> 00:29:14,760
pain um not in that application um to be

643
00:29:12,080 --> 00:29:19,120
honest um joints when you have lots of

644
00:29:14,760 --> 00:29:22,799
data probably using joints you know not

645
00:29:19,120 --> 00:29:26,919
always uh effective and and you still

646
00:29:22,799 --> 00:29:29,039
use uh joints within this uh domains of

647
00:29:26,919 --> 00:29:31,120
data right so you have one domain of

648
00:29:29,039 --> 00:29:35,159
generated data in which you can use

649
00:29:31,120 --> 00:29:37,240
joints um and um and then for all the

650
00:29:35,159 --> 00:29:39,159
analytics we never use the application

651
00:29:37,240 --> 00:29:43,559
databases so all the data is pushed to

652
00:29:39,159 --> 00:29:45,960
bigquery so in bigquery um it's a still

653
00:29:43,559 --> 00:29:47,840
all all data is presented together so

654
00:29:45,960 --> 00:29:52,279
for the analytics you can you can use

655
00:29:47,840 --> 00:29:52,279
jins um as as you want but

656
00:29:52,960 --> 00:29:59,320
yeah okay uh thank you very much um I'd

657
00:29:56,399 --> 00:30:03,679
like you to all uh help me to thank RTM

658
00:29:59,320 --> 00:30:03,679
for that uh excellent talk