perf: add mcache #46

vm-001 · 2024-10-10T16:48:39Z

Summary

Add MCache(multiple levels cache) to improve performance.

MCache Implementations

The MCache uses LRU as the L1 cache and redis as the L2 cache. To invalidate the L1 cache of different WebhookX nodes, it uses Notify / Listen to broadcast events.

codecov · 2024-10-10T16:57:15Z

Codecov Report

Attention: Patch coverage is 79.64912% with 58 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
db/dao/dao.go	68.51%	12 Missing and 5 partials ⚠️
eventbus/eventbus.go	71.92%	14 Missing and 2 partials ⚠️
app/app.go	60.60%	11 Missing and 2 partials ⚠️
mcache/mcache.go	80.32%	8 Missing and 4 partials ⚠️

Flag	Coverage Δ
integration	`66.67% <78.24%> (-0.89%)`	⬇️
unit	`14.30% <15.08%> (+0.55%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
config/config.go	`9.43% <ø> (-2.77%)`	⬇️
config/database.go	`76.92% <100.00%> (-7.70%)`	⬇️
constants/constants.go	`100.00% <100.00%> (ø)`
db/dao/attempt_dao.go	`100.00% <100.00%> (ø)`
db/dao/attempt_detail_dao.go	`89.65% <100.00%> (+0.76%)`	⬆️
db/dao/endpoint_dao.go	`100.00% <100.00%> (ø)`
db/dao/event_dao.go	`71.87% <100.00%> (+1.87%)`	⬆️
db/dao/plugin_dao.go	`100.00% <100.00%> (ø)`
db/dao/source_dao.go	`100.00% <100.00%> (ø)`
db/dao/workspace_dao.go	`100.00% <100.00%> (ø)`
... and 9 more

... and 48 files with indirect coverage changes

cchenggit · 2024-10-14T16:36:54Z

db/dao/dao.go

+		if e := mcache.Invalidate(ctx, key); e != nil {
+			dao.log.Warnf("failed to invalidate mcache: key=%s, %v", key, err)
+		}
+		dao.publishEvent(eventbus.EventInvalidation, map[string]interface{}{


use go routine to publish if not care returning

cchenggit · 2024-10-14T16:51:10Z

app/app.go

+	app.bus = eventbus.NewDatabaseEventBus(cfg.DatabaseConfig.GetDSN(), app.log)
+	app.bus.Subscribe(eventbus.EventInvalidation, func(data []byte) {
+		maps := make(map[string]interface{})
+		if err := json.Unmarshal(data, &maps); err != nil {


declare subscribe handlers somewhere else, keep app init code clean

SGTM. Updated.

cchenggit · 2024-10-15T16:08:08Z

config/database.go

-	)
-	return sql.Open("postgres", dsn)
+func (cfg DatabaseConfig) GetDSN() string {
+	dsn := cfg.DSN


Suggested change

dsn := cfg.DSN

dsn := cfg.dsn

cchenggit · 2024-10-15T16:08:44Z

config/database.go

+			cfg.Port,
+			cfg.Database,
+		)
+	}


Suggested change

}

cfg.dsn = dsn

}

cchenggit · 2024-10-15T16:09:04Z

config/database.go

 	"fmt"
 )

 type DatabaseConfig struct {
+	DSN      string


Suggested change

DSN string

dsn string

already have GetDSN() method, should keep DSN private.

I was going to add a new configuration field for the database section. For example:

database: dsn: postgres://username:[email protected]:5432/webhookx?sslmode=disable ...

The dsn will have the highest priority compared the the rest of all (username, password, etc)

deleted. (will be done on another PR)

cchenggit · 2024-10-15T16:28:05Z

eventbus/eventbus.go

+					handler(payload.Data)
+				}
+			}
+		case <-time.After(90 * time.Second):


DO NOT USE time.After in for select, it will cause memory leak, problem still exist in go 1.22

I think timer.After does not cause permanent leak, because GC will capture it when the underlying timer expires.

I replaced it by time.NewTimer() for the efficiency perspective.

cchenggit · 2024-10-15T16:43:42Z

mcache/mcache.go

+
+var defaultOpts LoadOptions
+
+func Load[T any](ctx context.Context, key string, opts *LoadOptions, cb Callback[T], id string) (*T, error) {


As opts could be nil, use ... Option chain to override defaultOpts for better experience.
for example:

func Load[T any](ctx context.Context, key string, cb Callback[T], id string, opts ...Option) (*T, error) {

Adding ... Option at the end of the function arguments make things a bit complex. I prefer to use an Options struct here.

db/dao/dao.go

Co-authored-by: webhookx-x <[email protected]> Signed-off-by: Yusheng Li <[email protected]>

webhookx-x · 2024-10-21T02:56:13Z

The eventbus can also be used for event router rebuild when source get changed.

vm-001 marked this pull request as draft October 10, 2024 16:48

vm-001 force-pushed the perf/mcache branch from ff74ad9 to 7bfe39b Compare October 10, 2024 16:54

vm-001 force-pushed the perf/mcache branch 2 times, most recently from 98fcbc1 to 10fd2c5 Compare October 12, 2024 03:07

vm-001 marked this pull request as ready for review October 12, 2024 14:04

vm-001 force-pushed the perf/mcache branch 3 times, most recently from 3889bab to fdd484d Compare October 13, 2024 02:24

vm-001 force-pushed the perf/batch branch from c5f5489 to 575766d Compare October 13, 2024 07:07

Base automatically changed from perf/batch to main October 13, 2024 07:56

vm-001 force-pushed the perf/mcache branch 4 times, most recently from bbfc603 to 6b0a054 Compare October 14, 2024 15:41

vm-001 requested a review from cchenggit October 14, 2024 15:41

vm-001 force-pushed the perf/mcache branch from 6b0a054 to 3d1a1dd Compare October 14, 2024 15:44

perf: add mcache

1d9251a

vm-001 force-pushed the perf/mcache branch 3 times, most recently from 7f89329 to fe38e83 Compare October 14, 2024 16:02

perf: add MCache

149ee16

vm-001 force-pushed the perf/mcache branch from fe38e83 to 149ee16 Compare October 14, 2024 16:07

vm-001 added this to the 0.3.0 milestone Oct 14, 2024

cchenggit reviewed Oct 14, 2024

View reviewed changes

refactor

cc494dc

cchenggit suggested changes Oct 15, 2024

View reviewed changes

vm-001 added 2 commits October 16, 2024 11:55

use timer to optimize memory usage

39aebf2

remove unused code

5a81792

webhookx-x requested changes Oct 21, 2024

View reviewed changes

db/dao/dao.go Outdated Show resolved Hide resolved

Update db/dao/dao.go

621b9a2

Co-authored-by: webhookx-x <[email protected]> Signed-off-by: Yusheng Li <[email protected]>

webhookx-x approved these changes Oct 21, 2024

View reviewed changes

webhookx-x merged commit 44453a0 into main Oct 21, 2024
6 checks passed

webhookx-x deleted the perf/mcache branch October 21, 2024 02:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: add mcache #46

perf: add mcache #46

vm-001 commented Oct 10, 2024 •

edited by webhookx-x

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

cchenggit Oct 14, 2024

vm-001 Oct 15, 2024

cchenggit Oct 14, 2024

vm-001 Oct 15, 2024

cchenggit Oct 15, 2024

cchenggit Oct 15, 2024

cchenggit Oct 15, 2024

vm-001 Oct 16, 2024 •

edited

Loading

vm-001 Oct 16, 2024

cchenggit Oct 15, 2024

vm-001 Oct 16, 2024

cchenggit Oct 15, 2024

vm-001 Oct 16, 2024

webhookx-x commented Oct 21, 2024


		var defaultOpts LoadOptions

		func Load[T any](ctx context.Context, key string, opts LoadOptions, cb Callback[T], id string) (T, error) {

perf: add mcache #46

perf: add mcache #46

Conversation

vm-001 commented Oct 10, 2024 • edited by webhookx-x Loading

Summary

MCache Implementations

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vm-001 Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webhookx-x commented Oct 21, 2024

vm-001 commented Oct 10, 2024 •

edited by webhookx-x

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

vm-001 Oct 16, 2024 •

edited

Loading